Track Record: 2017 Errors

This page first posted 22 June 2017, revised 24 June 2017

The headline prediction for the June 2017 election was not accurate. The final prediction was for a Conservative majority government with a majority of 66. The actual result was that the Conservatives were short eight seats of a majority.

In numerical terms, the prediction and the outcome were:

Labour's support was significantly underestimated, which caused the number of Labour seats also to be underestimated. Although the Conservative support figure was quite accurate, the error in the Labour support caused the predicted number of Conservative seats to be too high. The Liberal Democrats were also underestimated, but the prediction was accurate in saying that they would lose votes overall.

In Scotland, the prediction was correct that the SNP had passed its peak and would lose seats, mostly to the Conservatives. But the prediction did not foresee the full extent of the SNP decline. The predictions for the smaller parties were relatively good, with Plaid Cymru, UKIP and the Greens all predicted fairly accurately.

In total sixty-nine seats were mis-predicted. This would be an acceptable result if the general trend had been right, but the overall prediction quality was poor at this election. This was mostly due to polling error in the pre-election opinion polls.

We will now look at these and other issues in more detail. The particular topics studied are:

1. Confidence Bounds
2. Opinion poll error
3. Error Quantification
4. Model error

1. Confidence Bounds

The final prediction involved not just a central forecast, but also confidence bounds which the result should, with a 90pc confidence level, lie between. The confidence bounds given were:

The actual result was not close to the central forecast, but it was (mostly) inside the confidence bounds. The bounds were sufficient for all the parties, except for the Liberal Democrats who won four more seats than their upper bound despite losing vote share, and Plaid Cymru who got one more seat.

The confidence bounds were also shown in the final pre-election battleground graphic. The "Low-pollsters" were closer to the right answer than the "High-pollsters" and the "confidence circle" around them does include a sliver of "Conservative minority" territory. It was around this location that the final result materialised.

In summary, the confidence bounds worked fairly effectively and helped communicate in advance the relatively high degree of uncertainty about the result.

2. Opinion poll error

To make our prediction, we used an average of the final campaign polls taken by recognised polling organisations (members of the British Polling Council), plus a manual adjustment to correct for historic poll bias. This adjustment was defined as half the average error of polls over the last twenty years and was calculated as a swing of 1.1pc from Labour to Conservative.

There was a variety of values shown by the pollsters. Some were closer to the true result than others. Survation did particularly well with a poll showing a Conservative lead of 1pc, and Kantar Public who had a lead of 5pc. The other pollsters showed leads between 7pc and 12pc. The actual Conservative lead was 2.5pc, compared with the average over all the pollsters of 6.8pc. The Electoral Calculus adjustment for poll bias made our calculations worse, since the poll error was in favour of the Conservatives for the first time in many years.

The overall poll error in the Conservative lead was over 4pc. This is better than 2015 when the equivalent error was more than 6pc, and better also than 2001 and the bad error in 1992. But it was worse than in 1997, 2005 and 2010. So it was around the middle of the distribution of polling errors. The novelty this year was that the error overstated the Conservatives rather than, as is traditional, understating them.

In previous years, Electoral Calculus has also used spread betting market prices from Sporting Index. This year these were not used. On the eve of polling day, the seats market from Sporting Index showed the Conservatives with 363 seats and Labour with 205. This was less accurate even than the polls, so no accuracy was lost by the decision not to use it.

3. Error Quantification

Although poll error was the largest error, there were other errors. The main other errors were:

• Poll Bias correction. There was an Electoral Calculus adjustment of 2.2pc applied to the Conservative lead to correct for the polls' historic underestimation of the Conservatives. Since the polls overestimated the Conservatives this time, this adjustment was unhelpful and generated further error.
• EU Referendum Model. The Electoral Calculus model included voter transition behaviour based on past EU Referendum vote. In principle this should have increased Conservative votes in Leave areas and decreased them in Remain areas. Since this is what actually happened, this model feature should have improved accuracy. As it happens, the model did not behave in the correct way and made the overall prediction worse. This needs further investigation to determine the exact nature of the problem, but provisionally this is classed as another source of error.
• Tactical Voting. The final prediction did not include any adjustment for tactical voting as that had not been a significant feature in recent years and the old anti-Conservative tactical voting patterns seemed to have disappeared. However, there was significant anti-Conservative tactical voting in 2017. The omission of this feature was another cause of error.
• Residual Model Error. No model can perfectly predict all the seats, so there is always some residual model error once other effects are removed. This can be caused by idiosyncratic local behaviour, such as incumbency effects or other local issues, regional effects, or other non-uniform drivers of voter behaviour.

The table shows the effect of the errors as they are removed one by one:

Here "Error Count" is defined as the sum over the parties of the absolute difference between predicted and actual seats won. This is a larger measure than the number of seats mis-predicted.

We see that the largest reduction in seat error comes from the poll error which contributes a score of 50 out of the total error of 108. The bias correction was worth 14, the problematic EU Referendum model was worth 18, and the absence of tactical voting was worth a further 18. The residual model error produced an error score of 8, equivalent to a one seat error in predicting the Conservatives and Labour. That residual error is very acceptable and is well within the normal expected bounds for model error.

4. Seat by Seat errors

Even given all the corrections above, not every seat is predicted correctly. If we use the corrected parameters (removing the bias correction, fixing the pollster errors, remove the EU model and enable tactical voting) then we get the following result:

This is a much more accurate result overall, but the prediction is still inexact. In terms of individual seats, fifty seats are wrongly predicted, which is a only a fair result. In 2015 only 36 seats were mispredicted and in 2010 there were 63 mis-predicted seats. This suggests that the 2017 election had relatively high local variation, in that many seats deviated from the national trend and were either more Conservative or less Conservative than the model suggested.

[Note the use of Slide-O-Meter notation of "CON-03" to mean a Conservative majority of 3% which is used in this table. Majorities are rounded to the nearest integer percentage, so "CON-00" means a majority of less than 0.5%.]

There are a number of stories here. In outline they are:

• Anti-Con in London. There was a strong regional swing against the Conservatives in London which resulted in Labour picking up three more seats than expected [7, 8, 9].
• Scottish volatility. It was predicted that there would be a lot of close seats in Scotland, but there was quite a lot of volatility in both directions in safer seats. The SNP won 5 seats with unexpectedly large swings [23, 24, 26, 27, 30], but lost four others with similarly large swings [21, 22, 25, 28].
• Plaid Cymru strength in south-west Wales Plaid picked up two seats in south-west Wales with large unexpected swings [42, 43].
• Marginals. As usual there were a number of seats which were just marginal and could have gone either way. This year there were sixteen of them [1, 2, 12, 13, 14, 17, 18, 19, 20, 31, 32, 33, 38, 44, 48, 49].
• Local history. And there were some seats that had special circumstances or history. Brighton Pavilion [35] is a Green stronhold whatever the national polls say, Copeland [16] seemed to remember the recent Conservative by-election win, and Morley and Outwood [50] still fondly remembers dethroning Ed Balls in 2015.

Summary and Conclusions

• Our headline prediction for 2017 was not accurate
• The biggest cause of the error was poor estimation of national support levels by the pre-election polls conducted by the pollsters
• There were also errors caused by a misdirected "bias correction"; a questionable EU Referendum model; and a failure to predict anti-Conservative tactical voting
• Allowing for these known errors, the residual overall model error was small.
• But there was a high amount of local variation of seats around the model average.