Real Respondents

Sample Matters

There is a science to survey research – probability sampling. Probability sampling is the bedrock of reliable, accurate and projectable survey data.

With a probability sample respondents are real. They are selected from a well-defined universe with a probability that is known in advance. No other method does this.

As CNN said in their 2019 Polling Standards:

“Probability sampling …..remains the best place to start when it comes to polling methodology. It is still the gold standard.”

Many research companies have abandoned probability sampling or mix probability sample with unscientific sample to cut costs. There’s no need to sully survey data to make it affordable. The Modus way is scientific and cost-efficient. You don’t need to pay more for real data.

Real respondents means reliability

Unscientific sampling generates respondents that are motivated to earn modest financial rewards and so they do a lot of surveys. Such respondents will misrepresent themselves to qualify for surveys and are deeply affected by the learning effects of completing so many surveys.

Real respondents are mostly motivated by the opportunity to share their opinions. Without direct financial incentives (which Modus does not offer to its panel members), respondents lack the motivation to misrepresent themselves. That’s what makes them real.

Modus Research chooses all of its panel members randomly; respondents can’t join our panels without being invited via random probability telephone calls. They are real respondents.

A groundbreaking study conducted by the Market Research and Intelligence Association (MRIA) revealed that members of opt-in panels:

  • Are overwhelmingly motivated by financial reward 
  • Belong to many panels and complete many surveys 
  • Complete surveys at an alarming frequency and often speed through them

Unscientific opt-in panels are, in short, replete with professional (fake) respondents. Such respondents will often misrepresent themselves to earn modest financial rewards. Data from such panels are not reliable and should not be used to make important decisions.

With a Modus probability panel you get real respondents.

Probability means Known Data Accuracy

With a Modus probability sample you know the chance of a respondent being selected and so you know the accuracy of the data (i.e., the sampling error).

As the American Association for Public Opinion Research (AAPOR) task force stated in their 2013 report “Non-probability Sampling”:

 “AAPOR has long maintained reporting margin of sampling error with opt-in or self-identified samples is misleading”

Known population means survey results are projectable

We know the population from which we select our samples.

Knowing the universe and making sure that it is the group you are interested in is critical to being able to project survey results about that population. With unscientific sampling you simply cannot do this (although many claim they can). As AAPOR stated in their groundbreaking report:

“The dramatic rise in the use of opt-in panels has been premised on a willingness to accept overwhelming coverage and selection error.”

A Note on Weighting Schemes

Weighting the data does not overcome the issues with unscientific samples, no matter how exotic it may be. The Pew Research Center, a nonpartisan fact tank, concluded in their 2018 report: “For Weighting Online Opt-In Samples, What Matters Most?” that:

“… even the most effective adjustment procedures were unable to remove most of the bias [in opt-in panels]….. Even the most effective adjustment strategy was only able to remove about 30% of the original bias.”