MRG Evening Meeting 10 April 2001- The Response Rate Challenge
The first of four speakers addressing the issue of response rates in media research was Richard Asquith from BMRB who considers it to be the most important issue facing researchers today.
He began by clarifying that a response rate is the proportion of replies from a representative sample of eligible respondents from the survey universe. The reasons these can be difficult to attain are that people are either unavailable, refuse to participate or provide unusable data. Although sample sizes are key a poor response rate will lead to inaccuracies because if the sample is unrepresentative the audience measures will be distorted leading to wrong business decisions. There are also hidden problems due to the fact that it isn’t possible to calculate the response rate on most surveys.
Asquith used charts from high quality government surveys illustrating a gradual decline in response rates from the early 90s onwards. TGI managed to buck the trend recently by changing their incentives from a £3 voucher to £5 cash but this is very expensive and cannot be a long-term solution. Studies have identified four key factors in response rate influences: social context; respondent characteristics; interviewer attributes and survey design.
Having explained the challenge and identifying the factors affecting response rates, Asquith offered ideas as to what can be done. Good interviewer training must be offered; fieldwork must be managed well; there must be a good survey introduction; ensure the respondents can see the value of what they’re doing; give appropriate incentives and market the market research industry.
Researchers must make it easier to participate in surveys by providing data collection options to fit in with modern lifestyles, for example, by self-completion or online. Re-contacts and access panels can be used with care to provide additional information, as fresh samples are becoming progressively unrealistic. They must make participation in a survey a better experience by constructing surveys well and reduce the respondent burden by potentially splitting the research task and using data fusion.
Asquith concluded by stating that if media research is to ignore these issues it will lead to poorer quality research and therefore bad decisions. If they take these ideas on board however costs will increase and users must accept alternatives.
Katherine Page from IPSOS went on to explain how modern living is leading to a reduction in NRS response rates too because of factors ranging from increased lifestyle pressures; social attitudes; concerns over personal safety; image; negative experiences with previous surveys to concerns about privacy and entry phones. She argued that researchers must concern themselves with tackling the few elements within these factors they can affect.
Page divided non-respondents into refusals and non-contactables. It seems that it is the non-contactable group that is increasing and it is this group which tends to comprise of the younger and higher socio-economic groups while those that refuse to respond tend to be older and of a lower socio-economic group.
Refusals to respond can be tackled with improving interviewer skills, giving reassurances, improving the market research experience and providing incentives. Those that can’t be contacted need repeated calls, re-issues of surveys and different ways of gaining access must be considered if potential respondents are never at home. This problem is particularly acute in London.
The NRS are currently testing a mixed methodology using face-to-face interviews where possible as usual but also leaving self-completion questionnaires if not. These tests are proving positive and quantitative tests will start in May 2001 but the NRS still need to consider if they can increase response rates and most importantly, use the data.
Page concluded that these two types of non-respondents have different implications. Response rate strategies need to be targeted as response rate pressure may aggravate internal biases and researchers must have more flexibility in adapting to the respondent.
New MRG panel member, Lisa Beaumont from Taylor Nelson Sofres, addressed the particular challenges of recruiting and maintaining continuous panels, such as BARB and Superpanel.
She identified the keys in this area as firstly recruiting the panellists, then retaining them and finally ensuring their compliance.
Members of the Superpanel have their grocery consumption, media consumption, lifestyles and tv viewing analysed. They are recruited by invitation and, if accepted, telephoned, sent a recruitment package and are visited by an engineer if a tv setmeter is also to be installed. Although a lot is asked of Superpanel members there are relatively small recruitment targets.
Incentives are given in the form of thank-yous for compliance and ongoing contact through a newsletter. The aim of these is to improve rapport, maintain continuity and not to reward poor performance. Incentives are also targeted to specific audiences, for example offering virgin vouchers to attract under 27s improved the panel balance by 48%.
Beaumont argued that researchers should increasingly think of their respondents as customers and adjust their demands accordingly by providing incentives, easing the workload by reducing complexity and their time spent on the surveys.
Other issues include ensuring sample compliance, exercising quality control and weighting to bring the panel in line with known population profiles.
In conclusion Beaumont argues that as people live increasingly busy lives, researchers must make the ongoing task quick and simple, invest in the latest technology to ensure the quality of compliance and target incentives appropriately for hard to recruit demographic groups. The fact still remains that many respondents take pride in the contribution they make.
The final speaker of the evening was Rosemary Taylor from Rosemary Taylor & Associates concerned with the intriguing issue of Virgins, Groupies and Frauds. She addressed response rate issues of particular concern to those conducting qualitative research.
Taylor expressed concern at the growing level of ridicule targeted at qualitative research that is leading to distrust of its conclusions. She identified three areas where things can go wrong – miscommunication, inaccurate recruitment and fraud.
The Association of Qualitative Researchers is in the process of drawing up best practice guidelines, have considered a national respondent database and the drawing up of a respondent contract to create a culture of honesty and transparency.
The typical focus group is made up of virgins and groupies and dynamics are affected if there is a misbalance of these first time and experienced attendees. It has been found that the experienced respondent is less anxious and defensive, is happy to talk in front of others and is able to think less literally and more laterally. The positive effect of mixing virgins and groupies is that virgins may be more relaxed with groupies but then again, negatively, may give deference to those groupies among them. The very experienced groupie however seems to have few positive effects in that they can often become bored and surly, dominate the proceedings and tend to be strange and dysfunctional characters!
To conclude Taylor suggested that qualitative researchers need to think about the recruiter’s task; give her time to complete it; consider the amount of money paid to her; get out (of London) more and observe, without checking up on, proceedings. Qualitative research is not just group-discussion and the problems it faces with response rates are part of the issue facing market research.
Subscribers can access ten years of media news and analysis in the Archive