Number of U.S. vegans doubled in 2 years! Or not… (A note on interpretation of survey data)
February 25, 2012 § Leave a comment
Some vegan blogs have reported that a recent poll commissioned by The Vegetarian Resource Group and conducted by Harris International shows a dramatic growth in the number of vegans in America (“more than double the number of vegans in the U.S. since 2009“!). I would like this to be true as much as anyone, but the poll does not provide any basis for such a conclusion. Sorry to disappoint you.
It is possible that the number of vegans has, indeed, doubled since 2009. However, based on the results of the 2011 survey, it is just as likely that the number has fallen. Or has not changed at all. Here is why we should not use this survey to claim that the number of vegans has changed since 2009:
When we deal with figures that are very low to begin with (in 2009, roughly 1% of respondents reported following a vegan diet), even a twofold increase would be undetectable in a survey that looks at a sample rather than the entire population, because the change would fall well within the margin of error (plus or minus 3% in the 2011 survey). The 2011 survey pegs the number of vegans at 2.5%, which indicates no statistically detectable change compared to 2009. You will notice that The Vegetarian Resource Group’s announcement of the 2011 survey results does not say that the number of vegans in the U.S. has grown. The data simply would not support such a claim.
Here is a more detailed explanation of the statistics from the 2011 survey. In the methodology section of the survey report we read:
“In theory, with probability samples of this size, one could say with 95 percent certainty that the results for the overall sample have a sampling error of plus or minus 3 percentage points. There are several other possible sources of error in all polls or surveys that are probably more serious than theoretical calculations of sampling error. They include refusals to be interviewed (e.g., non-response), question wording and question order, and weighting. It is impossible to quantify the errors that may result from these factors.”
This means that in the best-case scenario, we could be 95% sure that any given poll number is accurate within a 7-point range (the reported number plus or minus 3 percentage points). So, if 2.5% of respondents report to be vegan, for all practical purposes it is just as likely that the real number is 1% (or even lower) or 5% – unless poll numbers are way off, which we can expect to happen 1 time out of 20. (This is what researchers mean by “95 percent certainty.”) Now, this best-case scenario is an ideal that few pollsters can ever hope to achieve. So, we have to lower our expectations even further (it is hard to tell how much) once we consider that some people who were randomly selected for the 2011 VRG survey did not respond (the number is quite high in most surveys), that the sample was not perfectly representative of the overall population (in this case, the pollsters “weighted” survey data for “age, sex, race, education, region, number of adults in household, and number of telephone lines . . . where necessary to bring them into line with their actual proportions in the population”), that the wording of the questions and other factors may have influenced respondents’ answers, and other sources of error.
Real sticklers will also note that survey design of the 2009 study was not as rigorous as in 2011, which makes any comparisons even more problematic. Here is what Harris International said about their 2009 survey:
Harris Interactive® fielded the study on behalf of The Vegetarian Resource Group from May 1-5, 2009, via its QuickQuerySM online omnibus service, interviewing a nationwide sample of 2,397 U.S. adults aged 18 years and older. Data were weighted using propensity score weighting to be representative of the the [sic] total U.S. adult population on the basis of region, age within gender, education, household income, race/ethnicity, and propensity to be online. Using traditional methods, with a pure probability sample of 2,397 adults, one could say with a 95 percent probability that the overall results have a sampling error of 2.7 percentage points. However, that does not take other sources of error into account. This online survey is not based on a probability sample, and therefore, no theoretical sampling error can be calculated. Nonprobability samples can still be representative of the population but cannot depend upon the rationale of probability theory (emphasis added).
By the way, bloggers are not the only ones who commonly misinterpret survey results. For a somewhat dated but still interesting example, see Robert Niles’s analysis of a media report on a presidential campaign poll.
Updated on March 5 and June 2, 2012