Why Sample Matters
Most people think that sample size is the most important dimension of survey data quality – the larger the sample, the better … or so goes the thinking. While sample size is important, how you sample – probability vs. non-probability – matters much more. Modus is Latin for method.
There is a science to survey research – probability sampling. Probability sampling is the bedrock of reliable, accurate and projectable survey data. Absent this critical dimension, the size of the sample does not matter.
Modus panels are developed and maintained using strict random probability sampling. Our panels do not contain the AI bots and survey ‘pros’ that plague opt-in panels – just real, live, randomly selected respondents. Sample matters.
Measurable Accuracy
A probability sample provides the critical data point for knowing the accuracy of your results: the chance of a respondent being selected for your sample. Only if you have this data point can you know the accuracy of the data (i.e., the sampling error) and only then can you confidently cite the margin of error.
This avoids the embarrassment of qualifying your results with the now common (some say, misleading) disclaimers such as: “for comparison, a probability sample … would have a margin of error of …”
This is not just our opinion; among a growing body of expert literature, it is a key finding from the American Association for Public Opinion Research (AAPOR) Task Force on Non-probability Sampling Report.
“AAPOR has long maintained reporting margin of sampling error with opt-in or self-identified samples is misleading.”
Projectability
Unlike non-probability sampling, we know the population from which we select our samples. The central importance of probability sampling is merely not calculating sampling error (or margin of error) per se, it is to be able to project the results to a known population with a known degree of statistical confidence. This cannot be done using non-probability sampling.
Knowing the population universe and having reliable data for it (e.g., the Census) is critical to reliably projecting survey results about that population. Despite what some vendors say, you simply cannot do this with non-probability sampling. In opt-in sampling, there is no way to know the population universe you are sampling from and there is no population data for whatever it is. As AAPOR stated in their groundbreaking report:
“The dramatic rise in the use of opt-in panels has been premised on a willingness to accept overwhelming coverage and selection error.”
Non-probability sampling relies on untested assumptions about how the sampled population relates to the broader population. Yet there is no way to know this – the sampled population, drawn as it is from open internet invitations, is not known.
The dangers of non-probability sampling were the focus of a recent research brief from NORC at the University of Chicago. This review cites one study using a well-known research firm showing that over 80 per cent of respondents were likely to have misrepresented themselves to to earn a financial reward.
“Nonprobability sampling works fundamentally differently. Participation depends on who happens to see an ad, access a rewards program, or click a link online. There’s no way to know who had the chance to participate, or who never saw the invitation at all. People who volunteer tend to share similar motivations or behaviors, making it impossible to verify whether they reflect the full population.”
Further, a 2020 review of the research on probability versus non-probability samples was conducted by a group of internationally renowned experts. The report details the ways in which non-probability samples are constructed using various quota and stratification techniques and the elaborate weighting often used. The evidence shows that these types of data manipulation rarely improve the accuracy of non-probability samples. Importantly, vendors of these samples are not transparent about their techniques and so they cannot be tested. They concluded: “Based on the accumulated empirical evidence, our key recommendation is to continue to rely on probability sample surveys.”
“For nonprobability sample surveys, the justification for expecting accurate measurements rests on untested modeling assumptions that are based on a researcher’s beliefs about the characteristics that make the sample different from the rest of the population …
The basic problem with these modeling assumptions remains that they cannot be tested.”
A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research – Journal of Survey Statistics and Methodology, Volume 8, Issue 1, February 2020, Pages 4–36.
