Development and assessment of a machine learning tool for predicting emergency admission in Scotland

James Liley, Gergo Bohner, Samuel R. Emerson, Bilal A. Mateen, Katie Borland, David Carr, Scott Heald, Samuel D. Oduro, Jill Ireland, Keith Moffat, Rachel Porteous, Stephen Riddell, Simon Rogers, Ioanna Thoma, Nathan Cunningham, Chris Holmes, Katrina Payne, Sebastian J. Vollmer, Catalina A. Vallejos, Louis J. M. AslettNpj Digit. Medicine 2024 · DOI

Emergency hospital admissions (EA) represent a significant challenge for healthcare systems worldwide, often leading to longer stays and higher costs compared to planned admissions. In Scotland, EAs account for more than half of all hospital stays. To address this, Scotland has utilized the Scottish Patients At Risk of Re-admission and Admission (SPARRA) score since 2006 to help clinicians identify high-risk patients for proactive primary care. However, the previous iteration (v3) relied on decade-old statistical techniques. This paper details the development and validation of SPARRAv4, a modernized machine learning-based risk prediction tool designed for the entire Scottish population.

Methodology and Data

The researchers transitioned from a single-model approach to an ensemble of machine learning algorithms, including Gradient Boosted Trees (XGBoost), Random Forests, Artificial Neural Networks (ANN), and Penalized Generalised Linear Models (GLM).

Overview of the SPARRAv4 development process, including the conceptual framework (a), population demographics (b), and the data processing pipeline (c).

A key innovation in SPARRAv4 is the use of topic modeling for electronic health records (EHR). To manage the sparsity of medical data—where thousands of codes exist but few apply to any single patient—the team employed a Latent Dirichlet Allocation (LDA)-style model. This approach treats medical codes as "words" to identify underlying health "topics" (e.g., respiratory or skin disease clusters), reducing data dimensionality while enhancing predictive accuracy.

The study utilized a massive nationwide dataset from Public Health Scotland, encompassing approximately 4.8 million unique individuals and 468 million health records from 2013 to 2018. This included inpatient/outpatient records, A&E attendances, mental health data, and community prescriptions.

Key Insights: The Impact of Age and Deprivation

Analysis of the model's predictions revealed critical non-linear relationships between demographics and health risk:

  • The Age-Risk Curve: Risk is high in infancy, drops during childhood, remains stable until age 65, and then rises exponentially.
  • The Deprivation Gap: Socioeconomic status, measured by the Scottish Index of Multiple Deprivation (SIMD), is a primary driver of risk. The study found that individuals in the most deprived 10% of the population face an emergency admission risk equivalent to being 10 to 40 years older than those in less deprived areas.

To ensure transparency, the team used Shapley values to decompose predictions, allowing clinicians to see exactly how much specific factors like age or deprivation contributed to an individual's risk score.

Results and Clinical Impact

SPARRAv4 demonstrated superior performance over its predecessor:

  • Improved Discrimination: The model achieved an AUROC of 0.799, compared to 0.781 for SPARRAv3.
  • Better Calibration: Predicted probabilities closely matched actual outcomes, a vital feature for clinical decision-making.
  • Resource Optimization: The authors estimate that SPARRAv4 could target 3,000 avoidable admissions while requiring interventions for 1,500 fewer patients than the previous system.

Furthermore, the model showed remarkable temporal stability, remaining effective for up to three years without retraining, although regular updates are recommended to account for evolving clinical practices.

Limitations and Future Directions

While SPARRAv4 is a significant advancement, it currently lacks primary care data (GP notes), laboratory results, and lifestyle factors like BMI or smoking status. Additionally, while the model is highly accurate at prediction, it does not imply causality; a high score identifies a patient in need of care but does not specify the underlying cause or the most effective intervention. Future work will focus on integrating more diverse data streams and addressing the "performative prediction" paradox, where successful interventions based on the model's output can paradoxically make the model appear less accurate in future training cycles.

Summary updated

BibTeX
@article{liley2024development,
  author = {James Liley and Gergo Bohner and Samuel R. Emerson and Bilal A. Mateen and Katie Borland and David Carr and Scott Heald and Samuel D. Oduro and Jill Ireland and Keith Moffat and Rachel Porteous and Stephen Riddell and Simon Rogers and Ioanna Thoma and Nathan Cunningham and Chris Holmes and Katrina Payne and Sebastian J. Vollmer and Catalina A. Vallejos and Louis J. M. Aslett},
  title = {{Development and assessment of a machine learning tool for predicting emergency admission in Scotland}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01250-1},
}
Figure