Enhancing the measurement of sentence severity to ensure the accuracy of the sentencing guidelines project: impact achieved in NCRM phases III and IV

This NCRM impact case study was written by Dr Jose Pina-Sánchez, of the School of Law at the University of Leeds.

Summary

England and Wales have embarked on a transformative process of sentencing reform through the design of sentencing guidelines. The guidelines seek to promote consistency by structuring the sentencing process, encouraging sentencers to follow the same approach. Guidelines, however, should not affect the severity of the sentences imposed. To ensure that is the case, the Sentencing Council for England and Wales has the statutory duty to assess the impact of their guidelines and reformulate those that appear problematic.

To undertake such evaluations, the Council developed an index of sentence severity, which allowed comparisons across sentence outcomes measured in different units (e.g. fines in pounds, custody in days). Through the project ‘Tackling Selection Bias in Sentencing Data Analysis’, led by Jose Pina-Sánchez, funded by the NCRM (phase III), and set up in collaboration with the Sentencing Council, the research team reviewed the validity of the index of severity operated by the Council. Problems of precision were identified (e.g. the index did not discriminate by types of suspended sentences), while its face validity was questionable (e.g. community orders were deemed almost as severe as short custodial sentences, while 20-year-long custodial sentences were deemed only three times as severe as a 1-year-long sentence).

Based on methods appraised through different NCRM events, the research team developed a new index of severity capturing all main sentence outcomes used in England and Wales, redressing the problems of validity detected. This new index was adopted by the Sentencing Council in 2019 and has been in operation in all their guidelines’ evaluations since. As a result, the accuracy of these crucial impact assessments has been significantly improved.

Impact

The main direct beneficiary is the Sentencing Council for England and Wales. Given the profound implications of its guidelines, and the ongoing incarceration crisis, the work of the Council has been intensely scrutinised. The first impact assessments of sentencing guidelines raised multiple questions, particularly in relation to the index of severity employed by the Council, which was criticised for being derived from a data-driven methodology. The adoption of the new scale of severity increased the transparency of the guidelines’ evaluations and confronted criticism directed at the reproducibility of the Sentencing Council’s analytical protocol.

The superior precision and validity of the new index has also allowed the Sentencing Council to increase the robustness of its impact assessments. Such evaluations are highly consequential since they serve to identify guidelines in urgent need of reformulation. By increasing the accuracy of their evaluations, the Council reduced the probability of incorrectly setting: i) well-functioning guidelines as ‘in need of reformulation’, and ii) guidelines that unduly affect sentence severity as if they were operating correctly.

By avoiding false positives, the new index has saved the Council time and resources associated to redesign well-functioning guidelines; and saved all sentencers and practitioners in England and Wales the necessary weeks of training needed to familiarise themselves with new guidelines. Yet, the greatest significance of the impact is in the higher sensitivity in detecting increases in sentence severity from an overuse of immediate custodial sentences. Improving the capacity to detect guidelines that are unduly inflating sentence severity, in turn, increases the speed with which the Council can reformulate these guidelines to redress the problem. This directly impacts the lives of offenders and their families, who would otherwise suffer the consequences of over-punitive sentencing. Furthermore, rectifying sentence-inflation benefits all British taxpayers by reducing public spending on prison beds.

Importantly, the impact achieved here has outlasted the duration of the research project and will be expected to influence the evaluation and design of sentencing guidelines until they either cease to exist or the analytical protocol employed by the Council is revamped in the future.

Role of NCRM

Beyond the direct funding provided to the project, the NCRM also played a key in perfecting its methodological approach. Pina-Sánchez and his team benefited from attending various of the NCRM methods courses taught by world renowned experts. These covered topics such as Measurement Error in Longitudinal Data (taught by Alexandru Cernat), or Bayesian Statistics (taught by Richard Morey). Some of the techniques learnt in these sessions were integrated in the methodology designed for the estimation of sentence severity using expert perceptions. For example, Bayesian methods were employed to check the robustness of the index of severity to potential sources of uncertainty like sampling error. Assessing this was key, as our index of sentence severity was elicited from a small sample of sentencing experts, which raised questions about its external validity.

In addition, Pina-Sánchez’s participation in two ESRC Methods Festivals, and another two events organised by NCRM, generated helpful feedback from both attendees and the directors of NCRM. Additional modelling approaches adopted in our index of severity were also revealed at some of the teaching activities organised by the NCRM. For example, we employed Thurstone scaling to reduce the cognitive burden of sentencing experts in their reports of the relative severity of different sentences, or Berkson measurement error models to reflect the fact that our estimates of severity are only approximations.

Since the completion of his project Pina-Sánchez still maintains a close relationship with NCRM. as demonstrated by his participation in the latest ESRC Methods Festival (held online in November 2021), where he delivered a session entitled ‘What Is Measurement Error (in in the Social Sciences)?’. He has also attended some of NCRM’s new short courses, covering topics such as ‘Reproducibility Protocols’. These courses are enabling him to put these methods into practice through his research, and in doing so to maintain the highest standard of methodological excellence in his ongoing collaborations with non-academic partners, such as the Sentencing Council for England and Wales and other Criminal Justice agencies, like the Crown Prosecution Service or the Parole Board.

In summary, by providing Pina-Sánchez with cutting edge methods resources, facilitating feedback from experts, and ultimately, funding the original project, the NCRM has, and continues to contribute significantly to the generation of the impacts described here. This represents a compelling case on how methodological proficiency can help improve public policy, and in so doing, enhance the common good.