Many people have noticed situations where the party forecast to win the seat does not have the largest probability of winning the seat. This paradox looks like there is an error in the calculations, make it is actually real.
We can understand what causes the paradox by looking at the example seat of Sheffield Hallam. On 26 May 2017, the seat prediction was as shown here:Sheffield Hallam
|2015 Votes||2015 Share||Predicted Votes|
|LIB Majority||2,353||4.2%||Pred Maj 0.9%|
|Chance of winning|
As we see, the Liberal Democrats were predicted to win the seat narrowly over Labour, but they were not the most likely party to win the seat. Labour had a higher chance of winning, and indeed its chance was more than 50pc. What accounts for this apparent discrepancy?
A key to unravelling the mystery is to understand how the prediction model works. The Strong Transition Model (STM) is more advanced than the Uniform National Swing model which assumes that all seats behave the same. With STM, voters can switch between parties using random distributions which are calibrated to the opinion polls. Additionally voters are classified as either "strong" (or sticky) voters who do not usually change party or floating voters who are more likely to change party.
But this means there is a complex response between a party's national vote share, and their projected vote share in a particular seat. We can see this for Sheffield Hallam:
The graph shows the vote share for the Lib Dems in Sheffield Hallam against their national vote share, plotted for 1,000 random simulations.
At the general election in 2015, the Lib Dems received 8.1pc of the national vote, but 40pc of the votes in Sheffield Hallam. Note that graph passes through that point – when the national vote share is around 8pc then the seat vote share is around 40pc.
The graph is curved, which is important and necessary. If the graph was a straight line, then worse paradoxes would arise. There are two relevant possibilities:
This corresponds to the basic UNS model, and means that for every 1pc increase (or decrease) in national support, the seat support also increases (or decreases) by 1pc.
The obvious problem with this model is when the Lib Dem national support is exactly zero, but the model still implies a seat support of 32pc although no votes were cast for the Lib Dems at all. The model also implies negative Lib Dem support in many seats to balance that. This is what mathematicians call an "infeasible" solution, and everyone else calls plain daft. It is not a good model, especially for declining parties.
This corresponds to a multiplicative model. If the national support doubles (or halves), then the seat support also doubles (or halves).
The problem with this model is twofold. Firstly, it is very harsh on declining parties and tends to predict they will get zero seats. Secondly it creates crazy results for gaining parties. If the Lib Dems got up to 20pc in the national polls, then the model predicts they would win more than 100pc of the votes in Sheffield Hallam. That is nonsense too.
Importantly, Labour's graph of votes in Sheffield Hallam is different.
On the whole, Labour's curve is a straight line. As a major party, the UNS model works less badly and the STM model fairly closely approximates a straight line.
The other crucial fact about the Lib Dem curve is that it is concave. This means basically that it has downwards-sloping curvature (Wikipedia: concave function)
And here is the crucial mathematical fact:
If one party has a concave response function, and another party has a straight line response function, and they have equal predicted vote shares, then the concave party has a smaller chance of winning the seat.
(This is a mathematical statement which is proved in the appendix, for those who are interested.)
This is the driver for the paradox. Because the Lib Dems have a convex response function, which is sub-linear, they lose out on average to the Labour party which has a linear response function.
The table above shows this result in the specific case of Sheffield Hallam. The x-axis along the top row represents Labour's national vote share which runs from 20pc to 50pc. The y-axis down the left-hand column has the Liberal Democrat national vote share, which can be between 0pc and 30pc. Each cell is coloured red or orange depending on whether the Sheffield Hallam seat is won by Labour or the Liberal Democrats. The number in the cell is the chance of it happening, given in hundredths of a percent. So a value of 50 means 0.5pc.
The total sum of the red cells is 4867, so Labour has a 48.7pc chance of winning. The total sum of the orange cells is 4283, so the Lib Dems have a 42.8pc chance of winning. This is an approximate calculation, but it shows that Labour has a higher chance of winning the seat. The predicted national vote shares were 32.5pc for Labour and 8.7pc for the Lib Dems. Using the table to look up this combination, confirms that the Lib Dems would win the seat if that is what happened. But the chance of that happening is less than the chance of Labour winning.
So the model is behaving correctly. As it is not possible for the Lib Dems to have a sensible linear response function, thus their concavity is natural, and the low win chance follows from that.
Let us start with some notation. Let:
Under the assumption that X and Y are jointly normal random variables, and that the seat is balanced so that f (E(X)) = g (E(Y)), then
P( f (X) > g(Y) ) < 0.5.
Let's define μX to be the expectation of X, so μX = E(X), and αX to be the gradient of f at μX, with αX = f '(μX). (If f is not strictly differentiable, then use the right-derivative.) Similarly, define μY = E(Y) and αY = g '(μY).
Since f is concave, it lies beneath its tangent at μX, so that
f (X) ≤ f (μX) + αX ( X − μX ),
with a positive chance of the inequality being strict. As g is affine, it equals its tangent at μY, so that
g (Y) = g (μY) + αY ( Y − μY ).
Thus the probability that party #1 wins the seat is
P( f (X) > g(Y) ) < P( αX ( X − μX ) > αY ( Y − μY ) ) = 0.5,
since f (μX) = g (μY), and X and Y are joint normals centred on μX and μY respectively.
Another factor which drives the paradox is the skew in the national vote share distribution. In practice this cannot be the normal distribution, because it the vote share must lie between 0 and 100pc. As described in this article, the random distribution used is a multi-variate beta distribution.
For the minor parties such as the Lib Dems and Greens, the beta distribution is skewed. That is, its mean is larger than its median. The mean is used for the central forecast, but the median is the driver of the win chances.
The graph shows the probability density function for the Lib Dem national vote share, along with the median and the mean of that distribution.
The fact that the median is less than the mean is another contribution to the lower win chances for Lib Dems.