Michael Marsh, Emeritus Professor, Trinity College Dublin, takes a look at what the polls got right - and wrong - in the run up to Election 2016.
Before all attention turns away from the election and towards to Dáil that has resulted it is appropriate to examine the record of the polls, and the Poll of Polls in particular and explore the "accuracy" of the final Poll of Polls and the predictions made on the basis of both the polls and the initial results.
I put accuracy in inverted commas here because the polls are a record of their time, and the final poll summary was published three days before the election. Since we do want to see those polls as a very good guide to the outcome it is fair enough to assess them in that respect, always allowing for the reason that the results and the polls were not perfectly aligned may be down to some systematic shifts in party support and not simply some methodological failings or oversights.
Before focusing on what the polls and the Poll of Polls got wrong it is worth saying that the polls were mostly right. If we had no knowledge at all from any polls I doubt that most people would have expected the government to have fallen so far from 2011, or that the newer parties could have been assessed anywhere near as accurately.
Even so, when we do look for "mistakes" there are some obvious differences between the Poll of Polls figures and those for first preferences in the actual election. These are most striking for Fine Gael and for Fianna Fáil. Fine Gael was overestimated by just over 3%, and Fianna Fáil was underestimated.
There was also an overestimation of the Sinn Féin vote, but this was smaller, less than 2%, and an underestimation of the Independent Alliance. In general the smaller parties and independents stood at 27.5% in the last Poll of Polls and won 29.1%, an underestimation of just 1.6%.
Arguably some of these small differences are within any accepted band of sampling error, and we might reasonably expect to make small mistakes in regard to the smaller parties and independents anyway as we know people who vote for these parties do tend to make up their mind to do so a little later than those who support the established parties.
However, the overestimation of Fine Gael and underestimation of Fianna Fáil are more serious. The expected gap between the two of over 7% turnout out to be just 1% on the day. What accounts for this?
The methodology of the Poll of Polls means that polls are weighted by time, with older polls weighted less – or excluded – as they are further from the time point of interest. It could be that there was a change in the last few weeks which was reflected insufficiently in the Poll of Polls. Indeed, it was because of this possibility that a separate calculation was included based just on polls in 2016.
However, looking just at the last poll conducted by each of the four companies who polls were included, the pattern of error is just the same. All companies were, it seems, too high for Fine Gael and too low for Fianna Fáil, although the extent of this varied across companies with IPSOS mrbi being closest to the final outcome.
Speculation as to why this occurred has been going on since the results were announced. Are people "shy" to admit they voted Fianna Fáil? (The same pattern of underestimating Fianna Fáil was also evident in the RTE exit poll in 2011 and 2016 and the Irish Times/Newstalk/IPSOS mrbi exit poll.)
If this is the explanation, why should "shy" Fianna Fáil voters be saying they would vote for Fine Gael – which has been the target of more than a little opprobrium in the media for a considerable time? An alternative possibility is a late swing against FG because of the leaders’ debates, or Enda Kenny’s comments about "whingers" for instance.
Late swing is a nice explanation because it can account for the underestimation of Fianna Fáil and overestimation of Fine Gael at the same time. There was also an upward trend for Fianna Fáil in the polls over the campaign but we have no hard evidence that this continued through the final week. The Exit poll did not show more Fianna Fáil voters (and fewer Fine Gael voters) among the late deciders. And of course late swing does not account for 2011, unless we have grounds for expecting that it happened in that election too.
The picture is made more murky when we look at the regional patterns for Fine Gael and Fianna Fáil. The underestimate by the 23 February Poll of Polls for Fianna Fáil was 4 points in Munster, 4 in Leinster, 2 in Conn/Ulster and just 1 in Dublin. These are not big differences but there are about 2000 respondents for each region so certainly the Munster-Dublin contrast is very unlikely to be an result of sampling error. If Fianna Fáil voters are ‘shy’ they are least shy where they are least prevalent, which is the opposite of what I would expect.
Fine Gael overestimates are a little more regular, but largest in Dublin (4), followed by Munster and Leinster on +3, and then Connacht/Ulster (1). In Dublin, the net beneficiaries from the Fine Gael overestimates were Others, in Munster is seems to have been Fianna Fáil, although in Munster Sinn Féin was also overestimated by 3 and Others underestimated by 2. Munster is where differences are greatest, with average poll-results differences averaging almost 3 percent across the four main parties and Others.
The RTE Exit poll 2016 also underestimated Fianna Fáil, suggesting 21 per cent and not the 24 per cent that party won. In contrast to earlier polls the underestimation was greatest – 4 points – in Dublin and Connacht/Ulster. The Exit poll also overestimated Sinn Féin, giving it 16 and not 14. Overall, however, the average error was just 1 per cent across these five party groupings. Further adjustment, once turnout levels were known, reduced that still further without changing the basic pattern.
More work will have to be done to identify the factors lying behind this apparently consistent pattern. Ironically, before the economic crisis the Fianna Fáil support was more likely to be overestimated, and some pollsters introduced adjustments to try to ensure Fianna Fáil support was not too high, a practice that was eventually abandoned as Fianna Fáil support went into freefall.
Making seat projections from national polls or vote shares is perhaps riskier than conducting a poll, though it is easier. I have shown the projections that I made in the table below, along with the (expected) actual result. With 10 seats still to be allocated final totals (as of 29 February) are uncertain but these are unlikely to be out by more than 1 either way. Predictions were made in two ways.
The first on the basis of the national picture, whether from polls, the exit poll or from the national share of the first count. I show the last here, since error will be down to the model that I chose to use and not to the data. The projections turned out to underestimate the Fianna Fáil seat total by 5 seats, and also gave three seats too many to the Social Democrats who won three and not 6, but elsewhere the fit is very good.
FG won a lot of seats with Labour transfers, and as the party did in 2011, it garnered a 33 per cent seat share that far exceeded its 25 per cent vote share. Some very effective vote management also delivered seats where they could hardly have been expected on the vote won. In past elections Labour had always done rather better than this method would have predicted, but this time it did not. Like most other small parties it did not win the share of seats corresponding directly to its share of votes.
The second estimation is based on the probabilities of each party winning a seat based on its first preference share at constituency level, and its hare of second preferences (taken from the Exit poll). These probabilities are derived from the statistical record of past elections, some 1000 cases of a party or group running in a constituency and exit polls back to 1997. The probabilities are then added up across all constituencies to give an estimate of likely national seat totals.
This approach gave a was very accurate prediction for Fine Gael ( just 1 seat out) and for the Social Democrats, as it took into account the fact that that its big votes at constituency level would not deliver two seats when the party was running only one candidate!
However, it overestimated Fianna Fáil seats by 4, and underestimated Sinn Féin by 3. All in all, each could be said to be equally accurate (or inaccurate) as in each case just 8/158 seats were misallocated. Neither method replaces Electronic Voting, or the need to listen though the night (or three) to get the final count from Dublin Bay North or wherever if you need to know as quickly as possible whether Labour will get enough seats to qualify to separate speaking time in the Dáil but of course is something is worthwhile it is worth making sacrifices for.
Michael Marsh, Emeritus Professor, Trinity College Dublin