UK flag US flag FR flag
| |

How did the polls get it so wrong?

How did the polls get it so wrong?

Why were the polls so wrong – and, given its reliance on data, should the media industry worry? We asked ZenithOptimedia’s research guru, Richard Shotton, to explain (spliced with some choice cuts from Twitter).

Samuel Goldwyn supposedly said “never make predictions…especially about the future”. After last night’s results many pollsters will be wishing that they had heeded his advice. But why were the pollsters so mistaken?

Firstly, it’s important to differentiate between the different types of poll. There’s a huge difference between the accuracy of the polls undertaken before voting and the exit polls. Whilst the opinion polls significantly under-estimated the gap between Labour and the Conservatives, the exit polls were remarkably accurate.

The initial exit polls estimated 316 Tory seats. When the final seats are announced they will probably be only be out by 10 or so seats.

Exit polls tend to be much more accurate. The 2010 exit poll predicted Tory seats exactly and got Labour and Lib Dems out by only three and two seats respectively. This higher degree of accuracy is due to a number of factors.

Firstly, the sample sizes are much bigger. Last night’s exit poll surveyed 22,000 people compared to most polls which survey about 1,400. Secondly, exit polls ask people what they have done rather than what they are planning to do.

Whilst people are uncomfortable lying directly about their past behaviour they’re notoriously inaccurate when predicting their own behaviour. There’s enough “wiggle room” in these self-predictions that biases can creep in.

One of the key biases that may have crept into the predictive polls is that of the social desirability. People, in both political and brand research, are reluctant to give answers which might reflect badly on them.

This translates into the “Shy Tory” effect where Conservative voters are less likely to declare their preferences for fear of being labelled uncaring. This bias was particularly apparent in the 1992 election where support for the Conservatives was predicted to trail Labour by 1% but in the end they secured an 8% lead.

Pollsters are, of course, aware of this bias and they take measures to account for it. Those who won’t state their preference are asked who they previously voted for. This historic behaviour is then used to weight the data.

Normally, this makes for a remarkably accurate series of predictions. However, re-weighting behaviour based on past patterns only works if the political environment is reasonably stable. Unfortunately this wasn’t the case in this election.

The nationalist surge, the punishment of a coalition partner and the rise of parties who previously had low support, like UKIP and the Greens – all these factors meant that precedents weren’t helpful.

The political landscape had undergone a step-change which meant past precedents were misleading.

Added to this is the fact that being accurate on a national level is less relevant than ever. If you compare the final poll of polls data with voting levels the predictions don’t look awful. Not great but not disastrous.

Conservatives did 3% better than the polls predicted , Labour 2% worse, UKIP / Lib Dems both correct, Greens out by 1 point (based on 638 seats).

However, the bigger issue is how these votes translated into seats. Perhaps one of the biggest changes to come will be that pollsters place more emphasis on seat by seat polling rather than national numbers.

What does this mean for the media industry?

I think we now have more evidence into the weakness of claimed data versus observed data. As an industry we’re remarkably reliant on the former (e.g. tracking data).

A few years ago this was acceptable because it was the least worst option. However, our opportunities are much more diverse now. We can access observed data (e.g. search data) which shows what people actually do is not what they claim. We can survey people on mobile at the point of purchase rather than on a PC in the office. Again this gets us much closer to the truth.

As an industry we need to move from claimed data (our equivalent to opinion polls) to observed and in situ data (our version of exit polls).

Nigel Jacklin, MD, Think Media Consultancy, on 08 May 2015
“Opinion polls are different to audience surveys in a number of respects.
1 Until you have voted your voting intention is an opinion; once you have voted it is a behavioural fact. Media research asks behavioural questions, so they are more like the exit polls.
2 Two thirds of people vote, one third do not. The pollsters need to 'predict' who will vote, so the accuracy of these predictions determine the accuracy of the polls. No equivalent adjustment is required for media surveys.
3 The first past the post system means that share of vote does not equate to electoral seats. In this election multi-party local situations were significant. These may be more complicated to reflect in the modelling.
For much 'every day' work we do as researchers the differences between the polls and the final results are within the normal tolerance limits.”

Media Jobs