The Inquiry set up to investigate inaccuracies in the opinion polls during the 2015 General Election campaign has released its preliminary findings which will be presented in detail today (Tuesday 19th January) at the Royal Statistical Society. The pre-election polls significantly under-estimated the size of the Conservative lead over Labour, with all the final polls suggesting a statistical ‘dead heat’.
Chaired by Patrick Sturgis, Professor of Research Methodology at the University of Southampton, the independent inquiry was set up last May by the British Polling Council (BPC) and the Market Research Society (MRS). It was tasked with establishing the extent of inaccuracies in the polls, the reasons for inaccuracies, and whether the polls were adequately communicated to the general public.
Following in-depth investigations, the Inquiry panel has concluded that the primary cause of the failure of the 2015 pre-election opinion polls was unrepresentativeness in the composition of the poll samples. The methods of sample recruitment used by the polling organisations resulted in systematic over-representation of Labour voters and under-representation of Conservative voters. Statistical adjustment procedures applied by polling organisations were not effective in mitigating these errors.
The conclusion that unrepresentative samples were the primary cause of the polling error is supported by comparisons of the polls with probability surveys conducted after the election (the British Election Study and the British Social Attitudes survey). These produced more accurate estimates of the lead of the Conservatives over Labour.
The Inquiry has also reached this conclusion by determining that other potential causes were likely to have made, at most, a modest contribution to the total error. These other factors include: voter turnout misreporting; question wording and ordering; treatment of overseas voters; treatment of postal voters; and treatment of un-registered voters.
The evidence on late swing – people changing their vote choice between the final polls and election day – is inconsistent. Several polls show some evidence of late swing to the Conservatives, while others show none. Given the differences in the methods used across polls, the inquiry has not been able to come to a definitive conclusion on whether late swing was a contributory factor in the polling miss. If late swing did occur, however, it only accounted for a small amount of the total polling error.
A surprising feature of the 2015 election was the lack of variability across the polls in estimates of the difference in the Labour and Conservative vote shares. Having considered the available evidence, the Inquiry has been unable to rule out the possibility that ‘herding’ – whereby pollsters made design decisions that caused their polls to vary less than expected given their sample sizes – played a part in forming the statistical consensus. It is important to note that the possibility that herding took place need not imply malpractice on the part of polling organisations.
The Inquiry will make a number of recommendations to the BPC and MRS when it publishes its report in March.
Professor Sturgis said, “There have been many theories and speculations about what went wrong in 2015 but, having considered the available evidence, the inquiry panel has concluded that the ways in which polling samples are constructed was the primary cause of the polling miss”.
ENDS
Contacts
For interview requests for Professor Patrick Sturgis please contact Peter Franklin, Senior Media
Relations Officer, University of Southampton: P.Franklin@soton.ac.uk 023 8059 5457/ 07748 321087
For information on the panel event taking place on the 19th of January please contact Magdalen
Wiley, Press & Policy Officer, Royal Statistical Society: m.wiley@rss.org.uk
Notes to Editors
1. The full details of the Terms of the Reference of the Inquiry are appended to this release.
2. Under the chairmanship of Prof. Patrick Sturgis, Director of the National Centre for Research
Methods at the University of Southampton, the Inquiry is charged with the task of establishing the
degree of inaccuracy in the polls, the reasons for the inaccuracies it identifies, and whether the
findings and conduct of the polls were adequately communicated to the general public. Due to report
in March 2016, the Inquiry is empowered both to make recommendations about the future practice of
polling and, where appropriate, for changes in the rules of the BPC. The BPC and MRS are committed
to publishing the Inquiry’s report in full.
In addition to Professor Sturgis, the Inquiry panel comprises eight people with professional
expertise and experience in conducting and analyzing survey
and polling data. They have received no remuneration for their work and none of them were directly
involved in conducting published polls during the election campaign. They are:
Dr. Nick Baker, Group CEO, Quadrangle Research Group Ltd
Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK
Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford
Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics
Prof. Jane Green, Professor of Political Science, University of Manchester
Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton
Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics
Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social
Research Association
3. The British Polling Council (BPC) is an association of polling organisations that publish
polls. The Council promotes standards of disclosure that are designed to provide consumers of
survey results that enter the public domain with an adequate basis for judging the reliability and
validity of the results. Most of the companies that conducted polls of voting intention at the 2015
UK general election are members. Further details can be found at
http://www.britishpollingcouncil.org/.
4. The Market Research Society (MRS) is the world’s leading professional research association,
training and regulating the research sector in the UK. The research sector is a major UK industry
worth a conservative
£3.6bn (GVA) per annum.
5. The University of Southampton connects with businesses to create real-world solutions to global
issues. Through its educational offering, it works with partners around the world to offer
relevant, flexible education, which trains students for jobs not even thought of. This connectivity
is what sets Southampton apart from the rest; we make connections and change the world.
http://www.southampton.ac.uk/ http://www.southampton.ac.uk/weareconnected
6. The Royal Statistical Society (RSS) founded in 1834 is one of the world’s most distinguished
and renowned statistical societies. It is a learned society for statistics, a professional body
for statisticians and a charity which promotes statistics, data and evidence for the public good.
http://www.rss.org.uk
7. The original announcement of the establishment of the Inquiry can be found at
http://www.britishpollingcouncil.org/general-election-7-may- 2015/.
BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election.
Terms of Reference
1. To assess the accuracy of the published opinion polls (both national and sub-national) at the
2015 general election.
2. To evaluate whether any inaccuracies identified might be part of a pattern evident at previous
elections.
3. To investigate the causes of any inaccuracies that are identified. Potential causes to be
considered will include (but not necessarily be limited to): the possible impact of late changes in
vote preferences, sampling methods, interview mode, weighting and filtering, population coverage,
item refusal, differential availability and willingness to participate, question order and wording.
4. To assess whether the analysis or reporting of polls was influenced by a reluctance to be out
of line with the published figures of other polls.
5. To consult and seek relevant evidence from all appropriate stakeholders, including but not
exclusively, polling organisations that are members of the BPC.
6. To assess whether adequate information was provided and communicated to interested commentators
and the public about how polls were conducted and what their results meant.
7. To make, as it sees fit, recommendations for improving how opinion polls are conducted and
published in future.
8. To make recommendations, if necessary, for changing the rules and obligations of BPC
membership.
9. To submit a report to the BPC and MRS by 1 March 2016, with a view
to its publication by BPC and MRS as soon as possible thereafter.
