Panels are Failing to Combat Fraud Effectively
Keith O'Brien
This is a series of blogs that explain how Realeyes’ Verify is the solution to fix a broken survey industry.
In our first post, we highlighted how there is a sharp increase in bad actors looking to exploit loopholes in survey collection. It’s a big business where fraud imperils a trillion-plus dollars in business decisions.
In this post, we’ll discuss what survey panels are trying to do to thwart these bad actors and why they’re falling short.
Until recently, survey fraud and the subsequent invalid data as a result were the industry’s dirty little secret. Only a few panels actually identified the fraud and put together an action plan to solve it (Kantar Profiles, a key Realeyes partner, being a leader in this regard).
A Kantar and Realeyes study of 200,000 recent survey sessions bought from the top three panel providers found some distressing statistics about survey panel fraud. For example, 27% of normal surveys contain over 20% duplicate users, with the duplication rate exploding to 153% for very large studies. In addition, 33% of surveys that pass current quality checks are bots or misreported demographics. In our first port, we identified the incentives for fraud. So, if it’s increasing, what are panels doing about it?
|
"Panel providers and suppliers have invested in all of the same deduping and quality controls - and yet, when I run my own quality assurance tools on top, I'm still getting rid of 10%-30% of poor quality sample. Clearly the panels aren't always enforcing quality, so we have to rely on our own toolsets," Scott Farrell, VP at Gazelle Global Research Services. |
There are two major issues with survey panels’ approaches to fraud, which we will explore in greater detail below. One, they have not necessarily been incentivized to address this fraud and, two, those who have confronted it have used mostly flawed tools. Even survey providers who utilize technologies to detect fraud are falling behind bad actors.
Missing incentives: Historically, the buy side (brands, agencies and market researchers) either doesn’t know how big of a problem fraud is, or don’t see any panel touting their solution, deeming better quality a non-starter. But since few were talking about it, the cost of addressing this fraud, when there were little competitive benefits, made committing to it a challenge.
Flawed tools: The goal of any anti-fraud tool is simple. Ban bad actors and let in those who are taking the survey honestly. Here are the major issues.
1. Failure to hit the sweet spot: Unfortunately, most of the tools on offer are rigid in some places and too strict in others. When companies are struggling to get enough respondents at an economical level, a high degree of false positives that filters out good respondents is unacceptable. One such tool, duplication detection, is geared to stop individuals from filling out multiple surveys from the same computer. But it is a blunt instrument: since it is based on device & IP addresses, it can create false positive dupes that don't account for multiple users in the same household using the same device.
2. Bots are dynamic: the tools are not. Anti-fraud technologies struggle to keep up with the sophistication of tools used by bots and sophisticated fraudsters. We have heard from multiple customers who have done forensic analysis / session playbacks to see one bot figure out the "solution" to get through, then propagate it to a host of other bots, all the while they seem like healthy machines / networks that are not in the lookup table. Another tool, Device / Network fingerprinting, is based on lookup tables of IP address & user agent strings that are stale. A panelist's system will appear infected, even though the bots moved on long ago.
3. The most accurate tool is not cost effective and limits survey participants. One way to guarantee a user is to use the Know Your Customer (KYC) framework utilized in financial service and other highly regulated industries where you must prove on a one-to-one basis individuals are who they say they are. It requires users to supply a government ID and a selfie, and the KYC identity vendors to validate the government ID and validate the selfie is the same person. It’s a heavyweight process for both panel and participant. And at a cost of $1+ per verification, the economics simply do not make sense for market research.
|
“Simply put, there is no silver bullet as nefarious actors work to outsmart multi-billion dollar brands and government agencies alike,” “Despite the multitude of fraud mitigation tools available to the industry, including behavioral analysis, digital fingerprinting, LLMs and more, collaboration is imperative for sample providers and buyers to effectively combat these challenges.” Lisa Wilding-Brown, InnovateMR CEO |
Surveys are created to be lightweight and easy enough for individuals to do. While most surveys compensate their users, there’s a tricky cost-benefit balance to make the time it requires to take survey is deemed an adequate trade-off to the compensation received.
If existing fraud detection tools are either not working or too cumbersome, what is the ideal solution? We will discuss that in our next topic.