The Internet is a research tool that offers speed and convenience. It offers immediate data capture and can potentially reach a more varied group of people than one could on foot. There is less reliance on the interviewer, but this also means that there are more factors a researcher cannot control, among these being response rates and whether or not your sample are in fact online. This is why a discussion of the validity of the medium in South Africa is a great way to understand its limitations and whether or not it’s the best way for any business to find the information it needs.
While we obviously believe in the power of the medium, it’s always important to look critically at what you’re trying to achieve. While Lyndi’s post offered some useful tips for ways you can use the Internet to test and gather information about people who are already online, we thought it useful to chat to someone who has been working with the gathering of primary research data.
Sarah Larter, Account Director at Millward Brown
Sarah is an Account Director at Millward Brown, one of the world’s leading research companies with offices in more than 43 countries. She specialises in quantitative research and works with clients to translate their business goals into research objectives.
As Account Director, Sarah has to set up a research strategy that will result in critically interrogated and therefore trustworthy results. In order to do this she needs to ensure a number of factors are in place and that she has access to specific kinds of information. She also has to determine which tools are best for finding the people she needs to reach.
As a tool that offers faster turn around times the Internet is understandably an attractive tool for both researchers and clients, and is thus increasingly used globally. “There’s nothing wrong with the face-to-face methodology, the big stumbling block is that it’s become inefficient.” This is why Millward Brown’s international offices focus so much on using digital tools.
“In the States I think 95% of the trackers that we do are based online. So people go online and answer a question online. There’s no face-to-face, there’s no knocking on people’s doors trying to find people.”
Despite its obvious benefits, however, Sarah has some criticisms of online as a research methodology in general and specifically in South Africa. Sample size, context and outside influences can all have an impact on whether or not information is truly insightful or just one person’s response.
While the Internet can, in theory, connect you with a huge sample, the first challenge is finding the right sample; not just anyone but the specific people you want to reach – the people who are relevant to your brand. The second challenge – and one we certainly face here in South Africa – is getting the number of responses you need to make a study valid. As a rule of thumb for getting a number that is representative of the audience, Sarah says that you'd need responses from about 100 people to get an answer that is statistically significant.
Response rates are incredibly low, which presents more of a challenge online. In a country like South Africa, where Internet penetration is below 25%, this means that the number of responses you get is often too small to be useful.
The final challenge, and quite a significant one, is that there’s a lot of information an online survey doesn’t supply.
“Errors creep in everywhere, our job as researchers is to minimise it … or at least to understand it so we can contextualise the findings in that light…with online you have even more checks to do and more bases to cover to make sure that you know what you are looking for, it can give you an answer – but what is that answer and what does it mean?”
The absence of the interviewer saves time but also means less control and information is available.
Sarah says that this by no means disqualifies the online medium, it just means you have to understand and carefully think through what you are asking and what the answers are really telling you.
The potential of mobile research is similar – an area where there is much more penetration in South Africa and Africa. While more people can respond much faster, you can’t ask as many questions as you could in a longer study. Again, this doesn’t mean it’s not useful, you just have to ensure you understand what you can gain from the medium and how best to contextualise your questions and your findings. FMCG brands, for example, where a faster turn around is essential, can benefit hugely from using mobile as a research tool.
Other online tools like social networks or online panels can also offer useful, if limited information. Practitioners in the UK and the States have had great success through setting up online chat rooms dedicated to researching a particular product. For researching existing products, crowdsourcing platforms like Dell’s Ideastorm offer a useful channel through which people can tell you what they do and do not like about a product.
These tools can all be used to great effect given that you understand what you are trying to find out. Online research can be and is an incredibly valuable tool for many situations - if you’re doing research about your online properties, again you can learn a lot from direct testing, the measurement tools are often built into the medium.
What I learned from Sarah Larter however, is that information is fragile and open to interpretation – to gain actionable insights you need to be sure you can trust it. If your respondents are online and do use the Internet it can save you time and money – just make sure your research is structured in a way that ensures the validity of your findings.
As Sarah said, quoting George Bernard Shaw, “The greatest problem with communication is the illusion that it’s been achieved.” Understand the medium and what you communicate as well as what is being communicated with you.
"The key thing to bear in mind is that face-to-face in home, telephonic, intercept interviewing, mobile and Internet are all methodologies - the route to the respondent. They all have pros and cons and so it's difficult to say outright that one is better than another. They all have there place. So in that context there is definitely a place for online. We just need to find it, define it and understand it."
This is obviously a huge topic, and one open to debate. What has your experience been and where do you see online research playing a role?
Did you enjoy this post? Why not check out:
As a retired interviewer with 20 years experience in Market research with a top UK company, l have taken part in almost every type of survey from hall tests to car clinics, most of my work however was social/political.
There is no such thing as a bad survey,--there are however factors which make a survey bad. Bad recruiting: that is when too many" interview whores" are in the sample, Market research companies will not want to talk about this however but it happens all the time. They need to supply the client with a sample so they cut corners and give interviewers a nod and a wink,also converting social class,age and working status , this is usually done by an executive under the pretext of a "back-check". On-line surveys are well known for this but it goes on every day with in- house and street interviews.
"Gold standard" surveys are with carefully selected "pannels" the respondents interviewed annually, this method is particulary accurate in testing trends over a period of time, the downside is price, as it's labour intensive with extra costs i.e incentive vouchers.
lastly online research, my advice to clients when contemplating this is obtain a copy of those who replied and do your own back-check, you are paying for this information,if you think that its a duff result, refuse to pay for it.
Posted by colin syme on 2009/04/24