Track Record: 2010 Model Comparisons

This page first posted 1 July 2010

This article looks at the official prediction model used for the 2010 election and compares it with other possible models. This will help show whether and how the official model could be improved for future elections.

The analysis is based on re-running the prediction, using both different models and different regional breakdowns. We will start be describing the models and breakdowns considered.

The sections are:

  1. Model Descriptions
  2. Regional Breakdowns
  3. Results
For an overview assessment of the 2010 election result, go to 2010 Track Record.

1. Model Descriptions

There are three main models considered, plus two variants of the third model.

Additive Model (UNS)

This is the simplest model in which a uniform additive swing is applied. In vector terms

SeatPredVoteShare = SeatVoteShare + AreaPredVoteShare - AreaVoteShare

The defect of this model is that the predicted vote share can go negative, or above 100%.

For details, see section 2 of Transition Model description.

Transition Model

This is a Markov-chain based model. Each voter makes a Markovian transition from the last election to the new state. Parties are divided into those "gaining" (predicted vote share is larger than actual), and those "declining" (vice versa). The model assumes: This model keeps the predicted seat vote shares feasible. But it can be overly pessimistic for declining parties.

For details, see section 3 of Transition Model description.

Strong Transition Model 20%

This is an enhancement of the transition model. Voters are divided into strong and weak supporters of each party. The strong supporters are those above the StrongThreshold, for example 20%. It is assumed initially that only weak supporters change sides. Only if a party's decline is more than the strong threshold, then strong voters will change sides.

Strong voters need to be identified seat-by-seat. By adding up strong votes over all the seats in the area, we can then calculate the AreaStrongShare. Note that this is the (turnout-weighted) average of the excess of SeatVoteShare over the threshold, which is larger than the excess of AreaVoteShare over the threshold [by Jensen's inequality].

This model keeps the advantages of the Transition Model, but produces more realistic behaviour for declining parties.

This is the current official prediction model. For details see Strong Transition Model description.

Strong Transition Model 25%

This is identical to the Strong Transition Model 20%, though using a strong threshold of 25% instead of the default 20% because this might be more accurate. We can look at the actual results in each seat and draw a plot of how the 2010 support for any party varies with its support in 2005 in the same seat.

Best fit coeffsSlopeOffset
Con < 25%1.09+1.8%
Con > 25%0.90+8.1%

Best fit coeffsSlopeOffset
Lab < 25%0.59+0.9%
Lab > 25%1.04-8.5%

Best fit coeffsSlopeOffset
LibDem < 25%0.87+3.4%
LibDem > 25%0.83+5.9%

We fit a best straight line independently in two regions: from 0% to 25%, and from 25% to 100%. We see that:

For the two major parties, the actual behaviour of seats is very consistent with the Strong Transition Model. Incidentally, the straight lines were the nearest to being continuous around a threshold of 25%.

Strong Transition Model 30%

This is also identical to the Strong Transition Model 20%, but using a strong threshold of 30% instead of the default 20%.

2. Regional Breakdowns

We also have six different ways of breaking down the country into separate pieces. Each piece is predicted separately, using the actual election vote shares for that particular piece. For finer breakdowns, the results will naturally be more accurate, but it is harder to estimate the vote shares before the election.

National (GB)

This is the simplest case, where we treat the whole country (excluding Northern Ireland) as a single entity. The prediction is based on the national vote share of the parties, using whichever model is selected.

Scotland

In this breakdown, seats are categorised as either "Scotland" (East and West Scotland regions), or "Not Scotland". The Scottish seats are predicted on the basis of actual Scottish vote share, and the non-Scottish seats are predicted on the basis of the actual non-Scotland vote share.

This is the breakdown which is generally used by Electoral Calculus, though not in the election campaign. The predicted vote shares are calculated from the Scotland-only polls and the national polls.

Areas

In this breakdown, seats are placed into their "area" categorisation. The definitions of the eleven areas are shown here. Each area is predicted on the basis of the vote share in that area.

This is the breakdown which was used by Electoral Calculus during the election campaign. The predicted area vote shares were calculated from YouGov regional polling. The quality of this was a bit patchy, see the regional section of the 2010 track record for details.

Geography

For this, we divide seats into three geographic categories which describe how rural or urban the seat is. The categories are:
Urban seats (256): Aberdeen North, Aberdeen South, Aldershot, Ashton under Lyne, Barking, Barnsley Central, Basildon and Billericay, Basingstoke, Bath, Batley and Spen, Battersea, Beckenham, Bedford, Bermondsey and Old Southwark, Bethnal Green and Bow, Bexleyheath and Crayford, Birkenhead, Birmingham Edgbaston, Birmingham Erdington, Birmingham Hall Green, Birmingham Hodge Hill, Birmingham Ladywood, Birmingham Northfield, Birmingham Perry Barr, Birmingham Selly Oak, Birmingham Yardley, Blackburn, Blackley and Broughton, Blackpool North and Cleveleys, Blackpool South, Blyth Valley, Bognor Regis and Littlehampton, Bolton North East, Bolton South East, Bootle, Bournemouth East, Bournemouth West, Bradford East, Bradford South, Bradford West, Brent Central, Brent North, Brentford and Isleworth, Bridgend, Brigg and Goole, Brighton Kemptown, Brighton Pavilion, Bristol East, Bristol North West, Bristol South, Bristol West, Bromley and Chislehurst, Broxbourne, Broxtowe, Bury North, Bury South, Camberwell and Peckham, Cambridge, Cardiff Central, Cardiff North, Cardiff South and Penarth, Cardiff West, Carshalton and Wallington, Castle Point, Chatham and Aylesford, Chelmsford, Chelsea and Fulham, Cheltenham, Chesterfield, Chingford and Woodford Green, Cities of London and Westminster, Colchester, Coventry North East, Coventry North West, Coventry South, Crawley, Croydon Central, Croydon North, Croydon South, Darlington, Denton and Reddish, Derby North, Derby South, Doncaster Central, Dudley North, Dudley South, Dulwich and West Norwood, Dundee West, Ealing Central and Acton, Ealing North, Ealing Southall, East Ham, Eastbourne, Edinburgh East, Edinburgh North and Leith, Edinburgh South, Edinburgh South West, Edinburgh West, Edmonton, Eltham, Enfield Southgate, Epsom and Ewell, Erith and Thamesmead, Exeter, Feltham and Heston, Finchley and Golders Green, Gateshead, Gedling, Gillingham and Rainham, Glasgow Central, Glasgow East, Glasgow North, Glasgow North East, Glasgow North West, Glasgow South, Glasgow South West, Gloucester, Gosport, Great Grimsby, Greenwich and Woolwich, Hackney North and Stoke Newington, Hackney South and Shoreditch, Halesowen and Rowley Regis, Halifax, Halton, Hammersmith, Hampstead and Kilburn, Harlow, Harrow East, Harrow West, Hartlepool, Havant, Holborn and St Pancras, Hornsey and Wood Green, Hove, Huddersfield, Hull East, Hull North, Hull West and Hessle, Ilford North, Ilford South, Ipswich, Islington North, Islington South and Finsbury, Kensington, Kingston and Surbiton, Kingswood, Leeds Central, Leeds East, Leeds North East, Leeds North West, Leeds West, Leicester East, Leicester South, Leicester West, Lewisham Deptford, Lewisham East, Lewisham West and Penge, Leyton and Wanstead, Lincoln, Liverpool Riverside, Liverpool Walton, Liverpool Wavertree, Liverpool West Derby, Luton North, Luton South, Manchester Central, Manchester Gorton, Manchester Withington, Mansfield, Middlesbrough, Mitcham and Morden, Motherwell and Wishaw, Newcastle upon Tyne Central, Newcastle upon Tyne East, Newcastle upon Tyne North, Newcastle-under-Lyme, Northampton North, Northampton South, Norwich North, Norwich South, Nottingham East, Nottingham North, Nottingham South, Old Bexley and Sidcup, Oldham West and Royton, Oxford East, Plymouth Moor View, Plymouth Sutton and Devonport, Pontypridd, Poole, Poplar and Limehouse, Portsmouth North, Portsmouth South, Preston, Putney, Reading East, Reading West, Rhondda, Richmond Park, Romford, Rotherham, Rutherglen and Hamilton West, Salford and Eccles, Sheffield Brightside and Hillsborough, Sheffield Central, Sheffield Hallam, Sheffield Heeley, Sheffield South East, Slough, Solihull, South Shields, Southampton Itchen, Southampton Test, Southend West, Southport, Spelthorne, St Albans, Stevenage, Stockport, Stoke-on-Trent Central, Stoke-on-Trent North, Stoke-on-Trent South, Stourbridge, Streatham, Sunderland Central, Sutton and Cheam, Swansea East, Swansea West, Telford, Thurrock, Tooting, Torbay, Tottenham, Twickenham, Uxbridge and South Ruislip, Vauxhall, Wallasey, Walsall North, Walsall South, Walthamstow, Warley, Watford, West Bromwich East, West Bromwich West, West Ham, Westminster North, Wimbledon, Wolverhampton North East, Wolverhampton South East, Wolverhampton South West, Worcester, Worthing East and Shoreham, Worthing West, Wythenshawe and Sale East, York Central.

Semi-urban seats (173): Aberavon, Airdrie and Shotts, Aldridge-Brownhills, Altrincham and Sale West, Alyn and Deeside, Amber Valley, Ashfield, Aylesbury, Ayrshire Central, Barnsley East, Barrow and Furness, Basildon South and East Thurrock, Beaconsfield, Bedfordshire South West, Blaenau Gwent, Blaydon, Bolsover, Bolton West, Bracknell, Burnley, Burton, Caerphilly, Calder Valley, Cannock Chase, Carlisle, Charnwood, Cheadle, Chester, City of, Chipping Barnet, Chorley, Christchurch, Clacton, Coatbridge, Chryston and Bellshill, Colne Valley, Crewe and Nantwich, Cumbernauld, Kilsyth and Kirkintilloch East, Cynon Valley, Dagenham and Rainham, Dartford, Derbyshire Mid, Derbyshire North East, Dewsbury, Don Valley, Doncaster North, Dorset Mid and Poole North, Dunbartonshire East, Dunbartonshire West, Dundee East, Dunfermline and West Fife, Durham North, Durham, City of, Easington, Eastleigh, Ellesmere Port and Neston, Elmet and Rothwell, Enfield North, Epping Forest, Erewash, Esher and Walton, Falkirk, Fareham, Filton and Bradley Stoke, Fylde, Garston and Halewood, Glenrothes, Gower, Gravesham, Great Yarmouth, Guildford, Harrogate and Knaresborough, Hastings and Rye, Hayes and Harlington, Hazel Grove, Hemel Hempstead, Hemsworth, Hendon, Hertford and Stortford, Hertfordshire South West, Hertsmere, Heywood and Middleton, Hornchurch and Upminster, Houghton and Sunderland South, Hyndburn, Inverclyde, Islwyn, Jarrow, Keighley, Kirkcaldy and Cowdenbeath, Knowsley, Lancashire West, Leigh, Llanelli, Loughborough, Makerfield, Meriden, Merthyr Tydfil And Rhymney, Middlesbrough South and Cleveland East, Milton Keynes North, Milton Keynes South, Morley and Outwood, Neath, Newport East, Newport West, Newton Abbot, Normanton, Pontefract and Castleford, Nuneaton, Ogmore, Oldham East and Saddleworth, Orpington, Oxford West and Abingdon, Paisley and Renfrewshire North, Paisley and Renfrewshire South, Pendle, Penistone and Stocksbridge, Peterborough, Pudsey, Rayleigh and Wickford, Redcar, Redditch, Reigate, Renfrewshire East, Ribble South, Rochdale, Rochester and Strood, Rochford and Southend East, Rossendale and Darwen, Rother Valley, Ruislip, Northwood and Pinner, Runnymede and Weybridge, Scunthorpe, Sefton Central, Shipley, St Helens North, St Helens South and Whiston, Staffordshire South, Stalybridge and Hyde, Stockton North, Stockton South, Stretford and Urmston, Surrey Heath, Sussex Mid, Sutton Coldfield, Swindon North, Swindon South, Tamworth, Thanet North, Thanet South, Torfaen, Tynemouth, Tyneside North, Vale of Clwyd, Wakefield, Wansbeck, Warrington North, Warrington South, Warwick and Leamington, Washington and Sunderland West, Waveney, Weaver Vale, Welwyn Hatfield, Wentworth and Dearne, Weston-Super-Mare, Wigan, Windsor, Wirral South, Wirral West, Woking, Wokingham, Worsley and Eccles South, Wrexham, Wycombe, Wyre and Preston North, York Outer.

Rural seats (203): Aberconwy, Aberdeenshire West and Kincardine, Angus, Arfon, Argyll and Bute, Arundel and South Downs, Ashford, Ayr Carrick and Cumnock, Ayrshire North and Arran, Banbury, Banff and Buchan, Bassetlaw, Bedfordshire Mid, Bedfordshire North East, Berwick-upon-Tweed, Berwickshire, Roxburgh and Selkirk, Beverley and Holderness, Bexhill and Battle, Bishop Auckland, Boston and Skegness, Bosworth, Braintree, Brecon and Radnorshire, Brentwood and Ongar, Bridgwater and West Somerset, Broadland, Bromsgrove, Buckingham, Bury St Edmunds, Caithness Sutherland and Easter Ross, Camborne and Redruth, Cambridgeshire North East, Cambridgeshire North West, Cambridgeshire South, Cambridgeshire South East, Canterbury, Carmarthen East and Dinefwr, Carmarthen West and Pembrokeshire South, Ceredigion, Chesham and Amersham, Chichester, Chippenham, Cleethorpes, Clwyd South, Clwyd West, Congleton, Copeland, Corby, Cornwall North, Cornwall South East, Cotswolds, The, Daventry, Delyn, Derbyshire Dales, Derbyshire South, Devizes, Devon Central, Devon East, Devon North, Devon South West, Devon West and Torridge, Dorset North, Dorset South, Dorset West, Dover, Dumfries and Galloway, Dumfriesshire, Clydesdale and Tweeddale, Durham North West, Dwyfor Meirionnydd, East Kilbride, Strathaven and Lesmahagow, East Lothian, Eddisbury, Faversham and Kent Mid, Fife North East, Folkestone and Hythe, Forest of Dean, Gainsborough, Gordon, Grantham and Stamford, Haltemprice and Howden, Hampshire East, Hampshire North East, Hampshire North West, Harborough, Harwich and North Essex, Henley, Hereford and South Herefordshire, Herefordshire North, Hertfordshire North East, Hexham, High Peak, Hitchin and Harpenden, Horsham, Huntingdon, Inverness Nairn Badenoch and Strathspey, Isle of Wight, Kenilworth and Southam, Kettering, Kilmarnock and Loudoun, Lanark and Hamilton East, Lancaster and Fleetwood, Leicestershire North West, Leicestershire South, Lewes, Lichfield, Linlithgow and East Falkirk, Livingston, Louth and Horncastle, Ludlow, Macclesfield, Maidenhead, Maidstone and The Weald, Maldon, Meon Valley, Midlothian, Mole Valley, Monmouth, Montgomeryshire, Moray, Morecambe and Lunesdale, Na h-Eileanan An Iar (Western Isles), New Forest East, New Forest West, Newark, Newbury, Norfolk Mid, Norfolk North, Norfolk North West, Norfolk South, Norfolk South West, Northamptonshire South, Ochil and South Perthshire, Orkney and Shetland, Penrith and The Border, Perth and North Perthshire, Preseli Pembrokeshire, Ribble Valley, Richmond, Romsey and Southampton North, Ross Skye and Lochaber, Rugby, Rushcliffe, Rutland and Melton, Saffron Walden, Salisbury, Scarborough and Whitby, Sedgefield, Selby and Ainsty, Sevenoaks, Sherwood, Shrewsbury and Atcham, Shropshire North, Sittingbourne and Sheppey, Skipton and Ripon, Sleaford and North Hykeham, Somerset North, Somerset North East, Somerton and Frome, South Holland and The Deepings, St Austell and Newquay, St Ives, Stafford, Staffordshire Moorlands, Stirling, Stone, Stratford-on-Avon, Stroud, Suffolk Central and Ipswich North, Suffolk Coastal, Suffolk South, Suffolk West, Surrey East, Surrey South West, Tatton, Taunton Deane, Tewkesbury, Thirsk and Malton, Thornbury and Yate, Tiverton and Honiton, Tonbridge and Malling, Totnes, Truro and Falmouth, Tunbridge Wells, Vale of Glamorgan, Wantage, Warwickshire North, Wealden, Wellingborough, Wells, Westmorland and Lonsdale, Wiltshire North, Wiltshire South West, Winchester, Witham, Witney, Worcestershire Mid, Worcestershire West, Workington, Wrekin, The, Wyre Forest, Yeovil, Ynys Mon, Yorkshire East.

Scotland/Geography

This is a coarse version of a breakdown by both location and urban/rural type. Seats are divided into four types, depending on whether it is Scottish and its geography (if it is not Scottish): This has the advantage that it could be practically possible to estimate the vote shares during an election campaign if a pollster chose to break down support by urban geography.

Area/Geography

This finely-sliced breakdown of the country categorises seats according to both their area and their urban/rural geography. Since there are 11 areas and 3 geographic types, there are up to 33 separate pieces. (In fact there are exactly 32 pieces, because there are no rural seats in the London area, but all other combinations are possible.)

This should give very good predictions, because of the fine level of detail. But it may be impractical before an election because of the difficulty in getting the predicted vote share in each of the 32 types of seat.

3. Results

We run the prediction separately for each of the five models and each of the six breakdowns described above. This gives 30 different predictions. For each prediction we compare it against the actual election outcome to see how accurate it is. Note that we are using the actual election vote shares to make the prediction, and we are not using the pollsters' estimated vote shares. This is to eliminate the pollster errors and help us to isolate the model error.

For any prediction, a simple measure of success is the number of seats which are mis-predicted. The final Electoral Calculus pre-election prediction mis-predicted 80 seats, and some of those were due to polling error. Adjusting for polling error, the remaining number of mis-predicted seats would have been 63. The number of mis-predicted seats for our thirty different model/breakdown combinations are shown in the table below:

AdditiveTransitionST 20%ST 25%ST 30%
GB7665737068
Scotland6565636260
Areas5764575455
Geography7365707065
Scot/Geog6067616164
Area/Geog5654555555

The best result is shown in bright green (54 mis-predicted seats), and predictions within 10 seats of that are shown in light green.

We note some conclusions:

A different measure of a prediction's success is just to look at the seat totals by party, and take the sum of the absolute differences with the actual result. Our final prediction scored 66 on this measure. Adjusting for polling error, this drops to 20. The table of error scores over all the combinations is:

AdditiveTransitionST 20%ST 25%ST 30%
GB3224262626
Scotland2440202024
Areas2017191915
Geography3022222224
Scot/Geog1424161614
Area/Geog1613131113

The conclusions above remain valid, although the urban/rural geography now is shown to cause a modest increase in accuracy.

Following these conclusions, we are overall content to keep using the Strong Transition Model, but with a revised threshold of 25%.


Return to track record summary, or home page.