The Inquiry set up to investigate inaccuracies in the opinion polls during the 2015 General Election campaign has released its preliminary findings which will be presented in detail today (Tuesday 19th January) at the Royal Statistical Society. The pre-election polls significantly under-estimated the size of the Conservative lead over Labour, with all the final polls suggesting a statistical ‘dead heat’.
Chaired by Patrick Sturgis, Professor of Research Methodology at the University of Southampton and director of the National Centre for Research Methods (NCRM), the independent inquiry was set up last May by the British Polling Council (BPC) and the Market Research Society (MRS). It was tasked with establishing the extent of inaccuracies in the polls, the reasons for inaccuracies, and whether the polls were adequately communicated to the general public.
Following in-depth investigations, the Inquiry panel has concluded that the primary cause of the failure of the 2015 pre-election opinion polls was unrepresentativeness in the composition of the poll samples. The methods of sample recruitment used by the polling organisations resulted in systematic over-representation of Labour voters and under-representation of Conservative voters. Statistical adjustment procedures applied by polling organisations were not effective in mitigating these errors.
The conclusion that unrepresentative samples were the primary cause of the polling error is supported by comparisons of the polls with probability surveys conducted after the election (the British Election Study and the British Social Attitudes survey). These produced more accurate estimates of the lead of the Conservatives over Labour.
The Inquiry has also reached this conclusion by determining that other potential causes were likely to have made, at most, a modest contribution to the total error. These other factors include: voter turnout misreporting; question wording and ordering; treatment of overseas voters; treatment of postal voters; and treatment of un-registered voters.
The evidence on late swing - people changing their vote choice between the final polls and election day – is inconsistent. Several polls show some evidence of late swing to the Conservatives, while others show none. Given the differences in the methods used across polls, the inquiry has not been able to come to a definitive conclusion on whether late swing was a contributory factor in the polling miss. If late swing did occur, however, it only accounted for a small amount of the total polling error.
A surprising feature of the 2015 election was the lack of variability across the polls in estimates of the difference in the Labour and Conservative vote shares. Having considered the available evidence, the Inquiry has been unable to rule out the possibility that ‘herding’ – whereby pollsters made design decisions that caused their polls to vary less than expected given their sample sizes – played a part in forming the statistical consensus. It is important to note that the possibility that herding took place need not imply malpractice on the part of polling organisations.
The Inquiry will make a number of recommendations to the BPC and MRS when it publishes its report in March.
Professor Sturgis said, “There have been many theories and speculations about what went wrong in 2015 but, having considered the available evidence, the inquiry panel has concluded that the ways in which polling samples are constructed was the primary cause of the polling miss”.
PowerPoint slides summarising the preliminary findings and conclusions are available from the NCRM website.