Opinion: with their sampling errors and shy voters, opinion polls are not perfect and come with an element of 'buyer beware' 

Opinion polls are an important part of the daily news cycle, as well as being a significant influence on decision making in many areas. An opinion poll, which is the systematic gathering of data about some issue, can influence stock markets, elections and referenda outcomes as well as the success or failure of products and services. 

Because of media's obsession with polls, they have now turned into a form of spectator sport. They've become a game to watch the rise and fall of opinion on newsworthy topics, whether politicians, political parties or issues and when there’s no real news, results of an opinion poll are handy to fill that gap.

Opinion polling is not a new activity and was first introduced in the 1830s in the United States. For about a century thereafter, polling was robust and reliable. Then, in the 1936 US presidential election, a large poll of more than two million respondents by the Literary Digest predicted that the incumbent Franklin D Roosevelt would lose the election to Alf Landon. Roosevelt won by a landslide and the explanation for the polling error was that the two million participants constituted a biased sample consisting of an over-representation of the more affluent members of society who would not have voted for Roosevelt. 

From New York Times, Nate Silver predicts the results of the 2012 US presidential election

In 2012, celebrated pollster Nate Silver almost perfectly predicted Barack Obama’s re-election and polling entered a golden few years of popularity and over-use. But pollsters got it all wrong, again in the 2016 US presidential election when Donald Trump defeated Hilary Clinton. Around the same time, pollsters also got it wrong with the predicted outcome of the UK Brexit referendum. 

In Ireland, Sinn Féin's rise in popularity and performance in the 2020 general election was not picked up in the opinion polls. Neither was the size of the majority in favour of the marriage referendum in 2015 nor the abortion referendum in 2018. The polling lessons from 1936 had Indeed been quickly forgotten. 

Today, polling has serious problems and its spectacular failures In the last half decade have been caused by two things: sampling errors and socially desirable responding, or shy voter syndrome. 

From RTÉ's The Week In Politics, a discussion on an Election 2020 opinion poll

Polling relies on a large or broadly representative sample of respondents in order to have a small margin of error and thus be useful and reliable. For decades polling was done by cold calling people on their landlines, but a 2019 survey found the response rate for telephone surveys to be only 6% and that's for the people who actually have landlines. 

More recently, polling has had to move online and some polling companies use survey panels, closed sets of respondents who agree to take multiple surveys on different topics. These are similar to an online version of our citizen's assembly where 100 carefully chosen citizens meet to consider questions and report on issues like abortion or population ageing.

Using polling survey panels could work reliably, but randomly selecting a broadly representative sample of respondents will vary from issue to issue. For example, a survey panel of people for polls on environmental issues may not be suitable for polls on LGBTQ issues

From BBC's Newsnight, did the pollsters get it so catastrophically wrong in 2015 because Conservative voters are just too shy to admit who they're voting for?

The second problem with modern polling is shy voters. Some respondents feel their real feelings and views on a topic may be socially unacceptable and so they give false answers when polled. This is especially true of sensitive issues like racism, immigration or same sex marriage, but it is also true of voting for Trump, Brexit or Sinn Féin. In each of these cases, the percentage who actually did vote was greater than the percentage who said they would when polled.

With their inherent flaws and reputational damage from recent polling disasters what is the future for this 190 year old practice of opinion polling? Polling must change and must do more to ensure their respondents represent a truly diverse and balanced range of views on each topic being polled.

One way to do this is to use implicit association tests (IATs). These are computer-based tests, first introduced less than 20 years ago, where the timing of their near-instant responses to rapid-fire word/picture associations can reveal respondents' inner views and natural predispositions on a topic.

From Serious Science, Prof. Mahzarin Banaji on implicit association tests

In some of our own work at the Insight centre in DCU, we used a panel of respondents and sought their views on the relationship between Ireland and the UK.  Working with colleague Dr Hyowon Lee and students Niamh Morris and David Hanley, we used traditional polling and IATs. No matter how we analysed results, we could not get IAT responses and traditional poll responses to strongly correlate. This tells us that, even within this small sample of dozens as opposed to hundreds of respondents, we see shy voter syndrome, even though all our tests were administered remotely and online.

When it comes to polling we need to be aware that polls are not perfect and that there is an element of "buyer beware" about them. While some media outlets like CNN have introduced polling standards, they do not address either of polling’s major problems. We hope those who rely on polls have learned from the past - or else we will once again be caught unawares over some future election or referendum result.


The views expressed here are those of the author and do not represent or reflect the views of RTÉ