Preventing Survey Fraud

There’s a simple and obvious answer to survey fraud. Probability sampling combined with a live telephone interview screens out professional respondents and inoculates our panels from A.I. bots. You cannot join a Modus panel unless we select you – randomly over the phone.

We identify survey pros (belong to two more more panels) at the moment of our intake interview. Through our probability-based recruitment of thousands of panelists, professional respondents make up less than 3% of the overall adult population – they are anything but representative. Our probability panels produce samples that have zero A.I. bots (they have no way of getting in) and we screen the pros who are motivated to commit survey fraud. As a final safeguard we offer no financial incentive to join our panels or complete surveys. 

The solution to rampant survey fraud is simply probability sampling. Sample Matters.

“… it’s structural. Nonprobability systems were built for speed and scale, not identity verification. When a system works not as invitation-only but as an open bar, fraud becomes a feature of the system, not a glitch.” 

David Dutwin, Executive Director and Senior Vice President at AmeriSpeak from NORC at the University of Chicago.

The Problem of Professional Respondents

Professional respondents are people who join multiple panels motivated to earn money by completing surveys. They join many panels (including the same panel multiple times by using various aliases and VPNs) and complete numerous surveys, often daily and hourly. You have to complete a lot of surveys to make money this way.

  • Professional respondents heavily populate opt-in panels.
  • They are real people but, based on our probability-based recruiting data, they make up less than 3% of the adult population. That is a very long way from representative.
  • Making them even less representative is the extremely high frequency at which they complete surveys. The learning effects are profound.
  • Opt-in panels explicitly advertise the opportunity to make money or ‘earn rewards’ by completing surveys. It only takes a simple google search to see this.
  • There’s many websites dedicated to helping people earn money by joining opt-in panels. One of the more popular of these websites, the ironically named Survey Police, shows that by qualifying for surveys targeting hard-to-reach targets (e.g., wealthy people, doctors, new mothers, business people) – hint, hint – “you can earn much more” money. These sites have virtually all of the industry-leading opt-in panels ranked on their pages (some even have affiliate links). 

This is how opt-in panels recruit their panelists. It is also how survey fraud became an epidemic in the Canadian market research industry. 

Financial incentives encourage survey fraud.

A Short History of Professional Respondents and Canadian Opt-in Panels - How Survey Fraud Became an Epidemic in Canada

The Canadian market is dominated by purveyors of opt-in panels. This dominance did not happen because they have superior data quality.

There has been a lot of pearl-clutching going on in the market research industry over survey fraud lately. They would have you believe that this is a recent phenomenon created due to the proliferation of A.I. bots.

While A.I. bots are indeed a problem for non-probability research panels, the core of the issue has been evident since the dawn of online research panels in Canada. 

In the summer of 2009, the now bankrupt and leading Canadian industry association – the Market Research Intelligence Association or MRIA – conducted a large-scale and groundbreaking investigation into the then recent proliferation of online research panels. This study involved eight leading research panels operating in Canada – seven non-probability panels and a lone probability one (although their report presented at the industry’s leading conference did not mention this critical distinction). The results were stark. They were also swept under the carpet.

The evidence of non-probability panels being overrun by professional respondents was clear. The implications of survey fraud were alarming:

  • In 2009 opt-in panels had an average of 47% professional respondents (using a very tolerant definition). 
  • On average, opt-in panelists belonged to 6.7 panels and 44% belonged to 5 or more panels. This was long before we had websites dedicated to promoting opt-in panels to make money.
  • Opt-in panelists were completing surveys at an alarmingly high frequency: two-thirds of opt-in panelists completed surveys at least once a week and almost 1 in 5 were doing them daily.  These numbers are certainly higher today.
  • Opt-in panelists are overwhelmingly motivated by financial rewards.

All of this was known by the Canadian industry in 2009. Yet the issue – non-probability sampling – was ignored by almost everyone. The opportunity to have a serious reflection on non-probability sampling was missed. Opt-in panels – heavily populated as they are with professional respondents – expanded and their merchants pressed on unhindered. 

Today, these merchants are all busily expounding on their advanced fraud and A.I. detection systems. Many articles and blog posts have been written – all necessitated by the original sin: non-probability sampling. 

The very short bar was the lone probability panel in the study.