Most people think that sample size is the most important dimension of survey data quality – the larger the sample, the better … or so goes the thinking. While sample size is important, how you sample matters much more. Modus is Latin for method.
There is a science to survey research – probability sampling. Probability sampling is the bedrock of reliable, accurate and projectable survey data. Absent this critical dimension, the size of the sample does not matter.
Modus research panels are developed and maintained using strict random probability sampling. Our panels do not contain the AI bots and survey ‘pros’ that plague opt-in panels – just real, live – randomly selected – respondents. Sample matters.
A probability sample provides the critical data point for knowing the accuracy of your results: the chance of a respondent being selected for your sample. You thereby know the accuracy of the data (i.e., the sampling error) and can confidently cite the margin of error.
This avoids the embarrassment of qualifying your results with the now common (some say, sketchy) disclaimers such as: “for comparison, a probability sample … would have a margin of error of …”
You do not have to trust us in saying this. It is a key finding from the American Association for Public Opinion Research (AAPOR) Task Force on Non-probability Sampling Report.
“AAPOR has long maintained reporting margin of sampling error with opt-in or self-identified samples is misleading.”
Unlike non-probability sampling, we know the population from which we select our samples. The central importance of probability sampling is not calculating sampling error (or margin of error) per se, it is to be able to project the results to a known population with a known degree of statistical confidence. This cannot be done using non-probability sampling.
Knowing the universe and having reliable population data for it (e.g., the Census) is critical to reliably projecting survey results about that population. With non-probability sampling you simply cannot do this (although many claim they can) – with opt-in sampling there is no way to know the universe you are sampling from and there is no population data for whatever it is. As AAPOR stated in their groundbreaking report:
“The dramatic rise in the use of opt-in panels has been premised on a willingness to accept overwhelming coverage and selection error.”
The AAPOR Task Force on Non-probability Sampling
Independent, third party research shows that non-probability sampling generates respondents that are motivated to earn modest financial rewards. This study was conducted by the now defunct Market Research and Intelligence Association (MRIA) and involved Canada’s leading opt-in panels and one probability-based panel. The results were at once shocking and stark.
Unlike probability sampling, opt-in panels consistently produce respondents that are:
The MRIA study revealed that non-probability opt-in panels are replete with professional respondents. Data from such panels are not reliable and should not be used to make important decisions.
Sample matters.