Garbage in, garbage out, goes the old saying. No doubt we’ve all heard that countless times during our market research careers. And nowhere is this adage truer than when considering the quality of your sample. What is quality when it comes to sample? You can’t really measure sample quality the way you can measure the quality of a physical product, or even a specific experience. While you may not be able to measure it absolutely, we believe it is imperative to be able to identify and avoid poor sample quality.
So, what are the most pressing considerations in assuring you don’t fall victim to investing in market research where poor online sample quality creates a flimsy foundation?
- Declining response rates due to lack of respondent engagement
- Respondent fears regarding their privacy and protection of their personal info
- The potential for respondents to misrepresent themselves and the emergence of survey bots
- The proliferation of panel sample providers with no industry standards for quality
High-quality marketing research agencies are aware of these potential pitfalls and work hard to mitigate or avoid them. Offering an engaging survey experience and being respectful of respondents’ time and effort goes a long way towards ensuring that our respondent pool will remain fresh and willing to respond to our requests for their opinions.
Creating standards of quality that must be met by all panel partners allows us to set clear expectations upfront and quickly weed out anyone who can or will not live up to those standards. And, developing internal quality checks to flag respondents who are “speeding” or “straightlining” through a survey (either because they are not engaged or because “they” are a “bot”) yields survey results in which we can have confidence.
There is, however, another issue that may not be so widely known (or acknowledged) and that is the fact that getting a sample representative of your target population does not “just happen” when you are using online panel sample.
It is commonplace to assume that any online sample you purchase – especially from a panel – is census representative. And the outgoing invitations very well may be. But, without careful thought and diligent execution, the incoming responses may be more representative of the sample management than anything else. For example, if you collect all your data in two or three days – or even less – you are likely to under-represent populations – like younger males – who tend to take longer to respond to surveys. Smart sampling requires taking steps to help ensure that individuals who are under-represented on panels or who are slower to respond always have a fair opportunity to participate in your survey.
In the same vein, if you don’t know the incidence of your target population or their demographic characteristics within the general population, research designed without a carefully crafted sampling management plan may yield results that lack statistical accuracy or reliability.
In many ways, sample quality is an invisible problem – until it becomes all too obvious in analysis or, worse yet, when you find that in-market performance does not reflect what your research indicated!
As larger and larger decisions and investments are being committed to based on marketing research, it is imperative that we protect its credibility by remaining vigilant about online sample quality. We would be wise to renew our respect for research as a science.