Track Record: 2019 Errors

This page first posted 27 January 2020

The headline prediction for the December 2019 election was broadly accurate. The polls fairly accurately indicated a substantial lead of the Conservatives over Labour, which our new regression-based model translated into a considerable Conservative majority. In the event the Conservative lead of Labour was slightly greater than the polls had shown, so that the Conservative majority was a bit bigger than expected.

In numerical terms, the prediction and the outcome for GB seats were:

Party2017 Votes2017 SeatsPred VotesPred Seats   Actual VotesActual Seats   Vote ErrorSeat Error
CON43.5%31843.3%35144.7%365−1.4%−14
LAB41.0%26233.9%22433.0%203+0.9%+21
LIB7.6%1211.7%1311.8%11−0.1%+2
Brexit0.0%03.2%02.1%0+1.1%0
Green1.7%12.7%12.8%1−0.1%0
SNP 3.1%353.6%414.0%48−0.9%−7
Plaid 0.5%40.4%20.5%4−0.1%−2
UKIP1.9%00.0%00.1%0−0.1%0

The final Electoral Calculus prediction is made up of two components: the poll of opinion polls and the seat predictor. Both parts performed well.

The poll-of-polls showed a Conservative lead over Labour of 9.4pc compared with an actual lead of 11.7pc. The difference between those, an error of 2.3pc, is low by historical standards. The equivalent lead errors were significantly higher in recent elections, with an error of 4.3pc in 2017 and a large error of 6.3pc in 2015.

Even with the poll error, the seat predictor performed very acceptably. The two major parties were within the margins of error, and the Conservative prediction of 351 seats was only 14 seats too low. Of all major UK seat predictors in 2019, the Electoral Calculus final prediction came closest to the actual result. Other predictors generally had the Conservative seat prediction too low, such as FocalData (337 seats), YouGov (339 seats) and Datapraxis (344 seats).

Of course, there is more to prediction accuracy than the headline seat count, since every seat is predicted individually which provides 650 opportunities to be right or wrong. Looking at the final Electoral Calculus seat-by-seat prediction, the winner was predicted correctly in 605 seats and incorrectly in 45 seats. That is a success rate of 93pc, which is the same as the high success rate of YouGov's final large-scale MRP in 2017. About a third of the incorrect seats were marginals which could have gone either way, leaving 31 seats which were mis-predicted.

We can also separate out the errors caused by polling and by the seat predictor. If we feed the actual national vote shares for GB and Scotland into the Electoral Calculus seat predictor we get the following result:

PartyOutcome
Votes
Predicted
Seats
Outcome
Seats
Seat
Error
CON44.7%362365−3
LAB33.0%2042031
LIB11.8%13112
Brexit2.1%000
Green2.8%110
SNP4.0%50482
Plaid0.5%24−2
UKIP0.1%000

You can see this for yourself by running the seat predictor with the correct national and Scottish vote shares: run predictor.

This is quite an accurate performance, with all parties correctly predicted to within just three seats. As it happens, this is as accurate as the TV Exit poll.

Of the two causes of error: polling error and predictor error, the polling error was actually the larger component, even though it was fairly small. The error in the prediction model was quite small indeed.

We will now look at these and other issues in more detail. The particular topics studied are:
  1. Northern Ireland
  2. Model errors
  3. Seat by seat errors

1. Northern Ireland

For the first time, Electoral Calculus was able to make predictions for Northern Ireland, in collaboration with our polling partners at LucidTalk.

The table below shows the predictions which were made and compares them with the actual outcome.

Party2017 Votes2017 SeatsPred VotesPred Seats   Actual VotesActual Seats   Vote ErrorSeat Error
DUP36.0%1030%1031%8−1%+2
SF29.4%725%623%7+2%−1
SDLP11.7%013%215%2−2%0
UUP10.3%011%012%0−1%0
Green0.9%00.1%00.2%0−0.1%0
Alliance7.9%016%017%1−1%−1
NI Other3.7%15%03%0+2%0

On the whole, the vote share predictions were fairly accurate. All parties were predicted to within 2pc of their actual vote share, which is within the margin of error (2.6pc) for a poll of 2,300 people. Our LucidTalk poll correctly predicted that the DUP and Sinn Fein would lose support compared with 2017 and that the Alliance would gain support. It also correctly predicted only small increases in support for the SDLP and UUP. In terms of seats, two seats were mis-predicted and sixteen seats were correctly predicted. Overall, the prediction was fairly accurate.

The two mis-predicted seats are described and explained below.

In Belfast North, we predicted a narrow DUP victory over Sinn Fein with a predicted majority of 3pc. In the event, Sinn Eein won the seat with a narrow majority of 4pc. Compared with our prediction, there is the suggestion of anti-DUP tactical voting by Alliance supporters, as the Alliance party polled 7pc lower than expected.

In Down North, we predicted a very narrow DUP victory over the Alliance with a predicted majority of just 0.2pc. The Alliance party actually won the seat with a majority of 7pc. This was a tricky seat to predict because the independent incumbent (Lady Sylvia Hermon) had stepped down and the UUP had not run in the seat recently. The UUP did better than we expected, and partially split the Unionist vote.

1.1 Northern Ireland methodology

This was the first time that Electoral Calculus made predictions for Northern Ireland and there was some interest from people in the province and elsewhere about how the predictions were made.

The first stage was a poll of 2,300 people in Northern Ireland conducted by LucidTalk and sponsored by Electoral Calculus and Remain United. LucidTalk performed their own analysis on the fieldwork data to calculate their estimates of the vote share for each party, as well as approximate probabilities for parties in each seat. At the planning stage, it was hoped to use regression-based techniques for these calculations. In the event, there was not enough time to perform a careful regression and classical polling methods were used instead.

You can see the LucidTalk tables for this poll on their website. The key figures are given in the spreadsheet of tables on the tab named 'Q2-WestminsterVote-ExcNVs' which has the following vote shares: DUP 30pc, SF 25pc, Alliance 16pc, SDLP 13pc, UUP 11pc, Other 5pc. These are the provincial vote share figures which drove the predictions.

LucidTalk also conducted their own accuracy analysis, focusing on a slightly different prediction, which was the vote share for the five large parties excluding smaller parties. They found, on that basis, that their vote share predictions were within 1pc of the actual result. (LucidTalk analysis).

The other key table was the LucidTalk NI Westminster Seat Predictor Model which is shown on page six of this LucidTalk document. For the record, those probabilities of victory in each seat for each major party are shown in the table below:

SeatPredicted
Winner
DUPSFAllianceSDLPUUP
Antrim EastDUPMax    
Antrim NorthDUPMax    
Antrim SouthDUP54% 26% 18%
Belfast EastDUP53% 42%  
Belfast NorthDUP49%46%   
Belfast SouthSDLP23% 19%55% 
Belfast WestSF Max   
Down NorthDUP47% 44%  
Down SouthSF Max   
Fermanagh and South TyroneSF 67%  28%
FoyleSDLP 45% 48% 
Lagan ValleyDUPMax    
Londonderry EastDUPMax    
Newry and ArmaghSF Max   
StrangfordDUPMax    
Tyrone WestSF Max   
Ulster MidSF Max   
Upper BannDUP Max   

In this table, the probabilities indicate the probability of the party winning the seat, not the predicted vote share. The entry 'Max' indicates the party has a very high chance of winning the seat, interpreted by Electoral Calculus to be around 80pc or higher.

The second stage of the process was conducted by Electoral Calculus. In this stage, predicted vote shares for each seat were inferred. The starting point for the optimization was the predicted results in each seat using the Electoral Calculus model applied to the LucidTalk provincial vote shares. These predictions were then adjusted using a search-based optimization to find adjusted vote shares for each seat which satisfy these three constraints:

  1. The total party vote shares over the province should match those calculated by LucidTalk
  2. The predicted winner in each seat should match LucidTalk
  3. the seat win chances should (approximately) match those calculated by LucidTalk

The results of this process were the predicted seat results shown on the Electoral Calculus website. In terms of the division of labour between LucidTalk and Electoral Calculus, LucidTalk was responsible for the province-wide vote shares and for the predicted winner of each seat; and Electoral Calculus was responsible for the predicted vote shares and for our seat win chances (which were similar to, but different from, the LucidTalk values due to modelling and other constraints).

1.2 Northern Ireland summary

Overall the Northern Ireland prediction was fairly accurate. The province-wide vote shares were within the margin of error for all parties. All but two seats were correctly predicted, and the two which were mis-predicted were correctly identifed as being very marginal.

Electoral Calculus is pleased to have partnered with our friends at LucidTalk on this project and to have benefitted from their well-informed judgements on Northern Irish politics.

2. Model errors

We can separate out the effects of polling error and model error. This is done by running the model using the actual national vote shares form the election and seeing how accurate the result is. This removes polling error because we are using the actual vote shares, and so the error that remains is model error.

This can be done both for the new regression-based model, which was the one used in the campaign, and also our older UNS-style strong transition model. This lets us compare between the two models to see if there is any noticeable difference.

The regression-based model used a regression-driven "baseline" prediction, which was then modified by a small UNS overlay to adjust for the difference between the regression's national vote shares and the target national vote shares. The regression baseline was based on two waves of campaign polling, each around 6,000 respondents in size.

The table below shows the actual election result in terms of both vote share and seats won; the UNS-style prediction using the actual vote share as the model input; and the regression-based prediction. The predictions are given in terms of seats and the error (defined as predicted seats minus actual seats) is also shown.

PartyActual
Votes
Actual
Seats
 UNS
Prediction
UNS
Error
 Regression
Prediction
Regression
Error
CON44.7%365351−14362−3
LAB33.0%203210+72041
LIB11.8%1117+6132
Brexit2.1%00000
Green2.8%11010
SNP4.0%4850+2502
Plaid0.5%43−12−2
UKIP0.1%00000

The number of incorrectly predicted seats in total was 48 for the UNS model and 34 for the regression model.

Generally the regression model performed better than the UNS model. The regression model predicted the total seats for each party to within a narrow tolerance of just three seats. But the UNS model was 14 seats too low for the Conservatives. (If the UNS model had been run with the predicted vote shares rather than the actual vote shares it would have predicted the Conservatives to win 344 seats, which is 21 seats too low.) The regression-based model was also more successful at predicting individual seats, having a seat error rate of 5.4pc compared with 7.6pc for the UNS-style model.

Overall the regression-based model was a noticeable improvement on the older UNS approach and was a more accurate predictor.

3. Seat by Seat errors

Even if we use the exact national vote shares, we do not predict every seat correctly. Although the total for each party is correctly predicted to within three seats, there are still 34 seats which were mis-predicted. This compares well with previous years. There were 50 seats wrong in 2017 (after correcting for polling and methodology issues), 36 in 2015, and 63 in 2010.

This is encouraging that the new regression-based model is having some success in converting national polling to lower-level geographies.

NumSeat NameGE2017PredictionGE2019AreaSwingComment
1MorayCON-09NAT-01CON-01Scotland1%Marginal
2Blyth ValleyLAB-19LAB-01CON-02North East2%Marginal
3DewsburyLAB-06LAB-01CON-03Yorks/Humber2%Marginal
4Dumfries and GallowayCON-11NAT-02CON-04Scotland3%Pro-Union
5DelynLAB-11LAB-04CON-02Wales3%Welsh anti-Labour
6Birmingham NorthfieldLAB-11LAB-03CON-04West Midlands4%Pro-Leave
7Banff and BuchanCON-09NAT-00CON-10Scotland4%Pro-Union
8Wolverhampton South WestLAB-05LAB-03CON-04West Midlands4%Pro-Leave
9Vale of ClwydLAB-06LAB-04CON-05Wales4%Welsh anti-Labour
10Colne ValleyLAB-02LAB-00CON-08Yorks/Humber4%Pro-Leave
11LincolnLAB-03LAB-02CON-07East Midlands5%Pro-Leave
12Stoke-on-Trent CentralLAB-12LAB-07CON-02West Midlands5%Pro-Leave
13Ynys MonLAB-14LAB-05CON-05Wales5%Welsh anti-Labour
14High PeakLAB-04LAB-09CON-01East Midlands6%Pro-Leave
15CheltenhamCON-05LIB-02CON-02South West7%Pro-Leave
16Norfolk NorthLIB-07LIB-01CON-28Anglia15%Incumbent steps down
17BedfordLAB-02CON-02LAB-00Anglia1%Marginal
18Coventry SouthLAB-17CON-02LAB-01West Midlands2%Marginal
19Alyn and DeesideLAB-12CON-02LAB-00Wales2%Marginal
20Coventry North WestLAB-17CON-03LAB-00West Midlands3%Pro-Remain
21HemsworthLAB-22CON-01LAB-03Yorks/Humber3%Anti-Con
22Sheffield HallamLAB-04LIB-06LAB-01Yorks/Humber4%Lingering Anti-Lib
23Wolverhampton South EastLAB-23CON-02LAB-04West Midlands4%Pro-Remain
24Bradford SouthLAB-16CON-04LAB-06Yorks/Humber5%Anti-Con
25HartlepoolLAB-18CON-04LAB-09North East7%Strong Brexit candidate splits Leave vote
26PutneyCON-03CON-09LAB-09London9%Surprise Labour win
27Dagenham and RainhamLAB-10CON-16LAB-01London10%Leave-ish incumbent holds on
28Portsmouth SouthLAB-03CON-03LAB-11South East12%Pro-Remain
29Fife North EastNAT-00NAT-04LIB-03Scotland8%Pro-Union
30St AlbansCON-11CON-07LIB-11Anglia14%Pro-Remain surprise
31CeredigionNAT-00CON-03NAT-16Wales2%Underestimated PC strength
32ArfonNAT-00LAB-04NAT-10Wales3%Underestimated PC strength
33Dunbartonshire EastLIB-10LIB-06NAT-00Scotland4%Surprise decapitation
34Kirkcaldy and CowdenbeathLAB-01LAB-28NAT-03Scotland11%Suspended SNP candidate wins anyway

[Note the use of Slide-O-Meter notation of "CON-03" to mean a Conservative majority of 3% which is used in this table. Majorities are rounded to the nearest integer percentage, so "CON-00" means a majority of less than 0.5%.]

There are a number of stories here. In outline they are:

You can also see the seat errors graphically. The graphic below is a "political triangle" which maps Conservative, Labour and Lib Dem support into a triangle, where each party has a corner of strength and is strong in points near that corner, and weaker for points further away. A party should win a seat if it falls inside the kite shape indicated by the internal dotted lines. (See seat migration for more details.)

Each seat is represented by a line and a blob. The line starts from the predicted outcome for the seat (using the regression model and the actual national vote shares) and finishes at the actual outcome for the seat, indicated by the blob. The colour of line indicates the predicted winner of the seat (blue for Conservative, red for Labour, orange for Lib Dem, green for Green, and yellow for SNP/Plaid). The colour of the blob indicates the actual winner of the seat.

You can hover over a blob to see the name of that seat. You can also press the button to toggle between viewing all seats and only those seats which were incorrectly predicted.

Outside Scotland and Wales, most of the incorrect seats were around the Labour/Conservative marginal border. Some of the strange seats, such as Putney, Portsmouth South and Hartlepool, are clearly visible as outliers on the "incorrect seats" view.

One clear pattern is that many seats near the Conservatives' border with the Lib Dems moved more towards the Lib Dems than expected. This suggests that anti-Conservative tactical voting was taking place, though there was not quite enough of it to win many seats, other than St Albans.

In the Labour corner, there was another pattern of strong Labour seats being even more stronger than expected. This was particularly noticeable in Labour cities, such as Liverpool, Manchester and parts of London. The Labour party platform and leadership was appealing to voters in those areas, while repelling more moderate voters elsewhere.

In the Conservative corner, there were a few seats which were even more Conservative than predicted. These were often in Essex and other Leaver strongholds, plus some home counties seats.

Summary and Conclusions

The main points that this analysis has shown are: The overall performance of the prediction was good, and the model was very good.
Return to top of page, or track record summary, or home page.