“Technology does not drive change, it enables change.” – Marshall McLuhan
Meaningful, lasting change must be conceived and propelled by leaders who understand how to harness the power of technology and use it to drive business in the right direction.
To fulfill our mission of being custom research consultants to our clients, RTi’s leadership team works to stay abreast of all new technologies and methods that impact our industry – and, where appropriate, uses this knowledge to offer our clients solutions that will help drive their businesses toward success.
Recently, a task force was created to look into Google Consumer Surveys to determine whether this was a platform that might be appropriate for some of our research initiatives. In short, the model Google has built is an alliance with 300+ online “periodicals” across the US who allow survey questions to be placed on their site. A reader can access limited free content on the site but must answer the pop-up survey in order to access the rest of the content. Originally the maximum number of survey questions was two – Google has very recently increased this to 10.
What we found is that while Google Surveys can be used for very basic or do-it-yourself survey research, it has some significant weaknesses and limitations as a serious tool for serious research. Specifically:
- Sampling Concerns: Because survey respondents are anonymous online “surfers”, Google does not have actual demographic information on people who complete any given survey. In place of this actual data, Google provides “inferred demographics”. RTi conducted a controlled test and found, for instance, that Google’s “inferred” gender of survey takers was correct only 74% of the time. Other inferred demographics were similarly off-base. This would not be an issue IF demographic questions could be asked in the survey – but a second limitation precludes this from being realistic…
- 10 Question Limit: If we cannot trust Google’s inferred demographic data, then we must explicitly ask these questions of survey participants. A typical demographic battery itself includes 6-8 questions, leaving only 2-4 questions available for surveying purposes and, therefore, making the 10-question model impractical. Even a shorter demo battery severely limits the number of survey questions that can be asked.
- Survey Chunking: Google’s contention is that instead of asking one group of people 50 questions, surveys can be “chunked” into 10-question increments, with five discrete groups of people each asked 10 of the questions. This model would severely restrict analytic flexibility – due to the limited number of questions each person is asked, something as simple as a driver analysis would produce results with considerably less statistical reliability than we would deem acceptable.
- Programming Limitations: There are too many of these limitations to list, but chief among them are the facts that only one screening question can be asked, size and content of images is restricted, and there are no “test links” available to check the survey prior to launch.
Given that this is Google’s first foray into the survey research arena, and given their resources, we’re confident that we’ll see improvements in their offerings over time. Therefore, we will continue to monitor advances in Google’s offerings, including a mobile platform that is rumored to be in the works. For now, however, it is clear that Google Consumer Surveys in its current form is not a strong or robust enough tool to support the custom, strategically focused, work RTi conducts on behalf of our clients.