Most people think that sample size is the most important dimension of survey data quality – the larger the sample, the better … or so goes the thinking. While sample size is important, how you sample matters much more.
There is a science to survey research – probability sampling. Probability sampling is the bedrock of reliable, accurate and projectable survey data. Absent this critical dimension, the size of the sample does not matter. Garbage in, garbage out.
Modus research panels are strictly based on probability sampling. As a result, they do not contain the AI bots and survey ‘pros’ that plague widely-used opt-in panels – just real people providing real answers. Sample matters.
A probability sample provides the critical data point for knowing the accuracy of your results: the chance of a respondent being selected for your sample. You thereby know the accuracy of the data (i.e., the sampling error) and can confidently cite the margin of error.
This avoids the embarrassment of qualifying your results with now common disclaimers such as: “for comparison, a probability sample … would have a margin of error of …”
You do not have to trust us in saying this. It is a key finding from the American Association for Public Opinion Research (AAPOR) Task Force on Non-probability Sampling Report.
“AAPOR has long maintained reporting margin of sampling error with opt-in or self-identified samples is misleading.”
Unlike non-probability sampling, we know the population from which we select our samples.
Knowing the universe and making sure that it is the group you are interested in is critical to reliably projecting survey results about that population. With unscientific sampling you simply cannot do this (although many claim they can). As AAPOR stated in their groundbreaking report:
“The dramatic rise in the use of opt-in panels has been premised on a willingness to accept overwhelming coverage and selection error.”
The AAPOR Task Force on Non-probability Sampling
Unscientific sampling generates respondents that are motivated to earn modest financial rewards and so they do a lot of surveys. Such respondents will misrepresent themselves to qualify for surveys and are deeply affected by the learning effects of completing so many surveys.
Real respondents are mostly motivated by the opportunity to share their opinions. Without direct financial incentives (which Modus does not offer to its panel members), respondents lack the motivation to misrepresent themselves. That’s what makes them real.
Modus Research chooses all of its panel members randomly; respondents can’t join our panels without being invited via random probability telephone calls. They are real respondents.
A groundbreaking study conducted by the Market Research and Intelligence Association (MRIA) revealed that members of opt-in panels:
Unscientific opt-in panels are, in short, replete with professional (fake) respondents. Such respondents will often misrepresent themselves to earn modest financial rewards. Data from such panels are not reliable and should not be used to make important decisions.
With a Modus probability panel you get real respondents.