By-elections are not good predictors

3 August 2023

Some readers of Electoral Calculus have written in recently to ask why we don't use by-election results in our predictions, or to demand that we should.

But historically, by-elections are a poor guide to the result at the next general election.

Let's look at a bit of history. Between 2010 and 2019, there were 36 by-elections. They are listed in the table below as a jog to the memory. The table shows the winner of the seat at the previous general election, the winner at the by-election, and the winner at the subsequent general election. The seven by-election gains, where the by-election winner is different from the previous general election winner, are highlighted in bold.

DateSeatPrev GEBy electionNext GEOutcome
1 Aug 2019Brecon and RadnorshireCONLIBCONOverturned
6 Jun 2019PeterboroughLABLABCON
4 Apr 2019Newport WestLABLABLAB
14 Jun 2018Lewisham EastLABLABLAB
3 May 2018Tyrone WestSFSFSF
23 Feb 2017Stoke-on-Trent CentralLABLABLAB
23 Feb 2017CopelandLABCONCONHeld
8 Dec 2016Sleaford and North HykehamCONCONCON
1 Dec 2016Richmond ParkCONLIBCONOverturned
20 Oct 2016WitneyCONCONCON
20 Oct 2016Batley and SpenLABLABLAB
16 Jun 2016TootingLABLABLAB
5 May 2016OgmoreLABLABLAB
5 May 2016Sheffield Brightside and HillsboroughLABLABLAB
3 Dec 2015Oldham West and RoytonLABLABLAB
20 Nov 2014Rochester and StroodCONUKIPCONOverturned
9 Oct 2014Heywood and MiddletonLABLABLAB
9 Oct 2014ClactonCONUKIPUKIPHeld
5 Jun 2014NewarkCONCONCON
13 Feb 2014Wythenshawe and Sale EastLABLABLAB
2 May 2013South ShieldsLABLABLAB
7 Mar 2013Ulster MidSFSFSF
28 Feb 2013EastleighLIBLIBCON
29 Nov 2012Croydon NorthLABLABLAB
29 Nov 2012MiddlesbroughLABLABLAB
29 Nov 2012RotherhamLABLABLAB
15 Nov 2012Cardiff South and PenarthLABLABLAB
15 Nov 2012CorbyCONLABCONOverturned
15 Nov 2012Manchester CentralLABLABLAB
29 Mar 2012Bradford WestLABRespectLABOverturned
15 Dec 2011Feltham and HestonLABLABLAB
30 Jun 2011InverclydeLABLABSNP
9 Jun 2011Belfast WestSFSFSF
5 May 2011Leicester SouthLABLABLAB
3 Mar 2011Barnsley CentralLABLABLAB
13 Jan 2011Oldham East and SaddleworthLABLABLAB
Source: Wikipedia, By elections 2010-present

Of the seven by-election gains, five of them reverted to the original party at the next general election. In other words, the by-election gain was temporary and the by-election result was not a good guide to the subsequent general election. There were two seats which bucked that trend, where the by-election gain was retained at the next general election:

We can see why by-elections are a poor guide to predicting general elections. Suppose we ignore national swings for a moment, and try to predict seats just on the basis of either the previous general election or by-election result. For the 29 seats where there wasn't a by-election gain, the two methods would give the same result and would correctly predict 26 out of 29 seats, with only three seats mis-predicted.

of seats
By-election gains which
revert back to the original party
By-election gains which
are held at the next GE

But for the seven seats which had a by-election gain, the two methods would be very different. Betting on the previous general election would be correct for five seats out of seven, and only wrong for two. But using the by-election result as a prediction would have the opposite result – with only two correct seats and five seats mis-predicted.

Going further back in time

We can go even further back in time and look at all by-elections since 1979. That's forty years of by-elections, and there were 166 by-elections in that time. Of those, most of them were held by the incumbent party, but there were 45 where the by-election winner was different.

Of these 45 by-election gains, the majority (26) were regained by the original incumbent party at the subsequent general election, and only 19 were held by the by-election winner.

The table below shows all 45 of those by-election gains from 1979 to 2019.

DateSeatPrev GEBy electionNext GEOutcome
1 Aug 2019Brecon and RadnorshireCONLIBCONOverturned
23 Feb 2017CopelandLABCONCONHeld
1 Dec 2016Richmond ParkCONLIBCONOverturned
20 Nov 2014Rochester and StroodCONUKIPCONOverturned
9 Oct 2014ClactonCONUKIPUKIPHeld
15 Nov 2012CorbyCONLABCONOverturned
29 Mar 2012Bradford WestLABRespectLABOverturned
23 Jul 2009Norwich NorthLABCONCONHeld
24 Jul 2008Glasgow EastLABSNPLABOverturned
22 May 2008Crewe and NantwichLABCONCONHeld
9 Feb 2006Dunfermline and West FifeLABLIBLABOverturned
15 Jul 2004Leicester SouthLABLIBLABOverturned
18 Sep 2003Brent EastLABLIBLIBHeld
21 Sep 2000Antrim SouthUUPDUPUUPOverturned
4 May 2000RomseyCONLIBLIBHeld
27 Feb 1997Wirral SouthCONLABLABHeld
11 Apr 1996Staffordshire South EastCONLABLABHeld
27 Jul 1995Littleborough and SaddleworthCONLIBLABOverturned
15 Jun 1995Down NorthUPUPUK UnionistUK UnionistHeld
25 May 1995Perth and KinrossCONSNPSNPHeld
15 Dec 1994Dudley WestCONLABLABHeld
9 Jun 1994EastleighCONLIBLIBHeld
29 Jul 1993ChristchurchCONLIBCONOverturned
6 May 1993NewburyCONLIBLIBHeld
7 Nov 1991LangbaurghCONLABCONOverturned
7 Nov 1991Kincardine and DeesideCONLIBCONOverturned
16 May 1991MonmouthCONLABCONOverturned
7 Mar 1991Ribble ValleyCONLIBCONOverturned
18 Oct 1990EastbourneCONLIBCONOverturned
22 Mar 1990Staffordshire MidCONLABCONOverturned
4 May 1989Vale of GlamorganCONLABCONOverturned
10 Nov 1988Glasgow GovanLABSNPLABOverturned
26 Feb 1987GreenwichLABSDPSDPHeld
8 May 1986RyedaleCONLIBCONOverturned
10 Apr 1986FulhamCONLABCONOverturned
23 Jan 1986Newry and ArmaghUUPSDLPSDLPHeld
4 Jul 1985Brecon and RadnorCONLIBLIBHeld
14 Jun 1984Portsmouth SouthCONSDPCONOverturned
24 Feb 1983BermondseyLABLIBLIBHeld
28 Oct 1982Birmingham NorthfieldCONLABCONOverturned
3 Jun 1982Mitcham and MordenLABCONCONHeld
25 Mar 1982Glasgow HillheadCONSDPSDPHeld
26 Nov 1981CrosbyCONSDPCONOverturned
22 Oct 1981Croydon North WestCONLIBCONOverturned
9 Apr 1981Fermanagh and South TyroneIndependentAnti H-BlockUUPOverturned
Source: Wikipedia, By elections 1979-2010

Most by-election gains were overturned at the subsequent general election. This means that purely using the by-election result as a predictor would be worse than just using the previous general election result as a predictor.

But the fraction of by-election gains which are retained, which is 42%, is not that small. That suggests a hybrid approach where we could take, say, the average of the previous general election result with the by-election result. If we do that on the 45 by-election gains from the historical data, then the hybrid predictor is correct 30 times out of 45. And that's a bit better than just using the previous general election, which was correct 26 times.

Prediction MethodCorrect predictions%
Previous general election result26 / 45 58% 
By-election result19 / 4542%
Average of previous GE and by-election30 / 4567%

The message of these data is clear. Both in the short-term and long-term, by-election results are not a great guide to the result of the next general election. More often than not, by-election gains are overturned at the next general election. But taking a mixture of the previous general election and the by-election result could be a better predictor than either on its own.