TL;DR
This paper introduces a machine learning model to predict sepsis 6 hours in advance in general hospital wards, outperforming traditional scoring systems. A novel pseudo-prospective trial design was used to better estimate real-world performance, showing promising results for early sepsis detection in a challenging clinical environment.
What Problem Does This Paper Address?
Sepsis is a life-threatening condition with high mortality rates, making early detection and treatment crucial. While many prediction models exist, they often focus on data-rich environments like the Emergency Department (ED) or Intensive Care Unit (ICU). General hospital wards, however, present a unique challenge due to sparser data and less frequent patient monitoring. Patients developing sepsis in these general wards often have worse outcomes, highlighting the need for effective early warning systems in this setting. This paper aims to bridge this gap by developing and evaluating a sepsis prediction model specifically for general wards.
Methods
The researchers employed a retrospective analysis of electronic health record (EHR) data to develop their sepsis prediction model. They utilized an eXtreme Gradient Boosting (XGBoost) model, a powerful and widely used machine learning algorithm for classification tasks. To make the model more interpretable and potentially easier to deploy, they also developed a "lite" version using a subset of the most impactful features identified by SHapley Additive exPlanations (SHAP) values. This approach helps understand which features contribute most to the model's predictions. For comparison, they also trained a logistic regression model and evaluated traditional scoring systems like SIRS, NEWS2, and qSOFA.
A key methodological innovation is the pseudo-prospective trial design. Instead of a single prediction per encounter, the model was applied hourly to patient encounters in the test set. This simulated a real-world scenario where continuous monitoring and alerting would occur, providing a more realistic estimation of the model's implemented performance.
Datasets
The study used a large dataset from the Electronic Health Record (EHR) data from Barnes-Jewish Hospital / Washington University School of Medicine in St. Louis. This dataset comprised adult, non-surgical inpatients admitted between January 1, 2012, and June 1, 2019. The final cohort included 70,034 patient encounters. The data included demographics, medications, vital signs, and laboratory results.
Key Insights and Ideas
- General Wards are a Critical Blind Spot: The study emphasizes that general wards, despite their data sparsity, are a crucial area for sepsis prediction due to the vulnerability of patients and worse outcomes compared to ICU/ED settings.
- Machine Learning Outperforms Traditional Scores: The developed XGBoost model significantly outperformed traditional scoring systems (SIRS, NEWS2, qSOFA) in predicting sepsis, demonstrating the power of machine learning in this context.
- Interpretability Matters: The use of SHAP values to identify key features and the development of a "lite" XGBoost model highlight the importance of not just predictive accuracy but also model interpretability and deployability in clinical settings.
- Pseudo-Prospective Trials Offer Realistic Evaluation: The novel pseudo-prospective trial design provided a more accurate reflection of how the model would perform in a real-world, continuous monitoring scenario, addressing limitations of purely retrospective evaluations.
- Time is a Crucial Predictor: "Time from admission to prediction time" was identified as the most impactful feature, underscoring the link between prolonged hospital stays and increased sepsis risk.
Key Results
- The optimized XGBoost model achieved an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.862 ± 0.011 and an Area Under the Precision-Recall Curve (AUPRC) of 0.294 ± 0.021.
- This performance was superior to logistic regression (AUROC: 0.857 ± 0.008, AUPRC: 0.256 ± 0.024) and traditional scoring systems like NEWS2 (AUROC: 0.699 ± 0.012, AUPRC: 0.092 ± 0.009).
- In the pseudo-prospective trial, the model achieved 69.7% sensitivity for alerting on septic patients with 81.4% specificity.
- Within 48 hours of an alert, 29.1% of alerted patients experienced a sepsis-related intervention or outcome.
Limitations and Future Work
The authors acknowledge several limitations:
- Single-Institution Study: The findings may not generalize to other healthcare systems with different patient populations or EHR data structures.
- Sepsis-3 Definition: The study used a specific interpretation of the Sepsis-3 definition, which might not align with other clinical definitions.
- Exclusion Criteria: While necessary to focus on patients who could benefit from the model, the exclusion criteria (e.g., excluding surgical patients) limit generalizability.
- Pseudo-Prospective Trial Nuances: The pseudo-prospective trial, while innovative, still has limitations. It did not account for repeated alerts, alert lock-out periods, or measurement-to-documentation time gaps. The chosen threshold to maximize F1 score might also need further operational refinement.
- Data Quality: The assumption of "normal" for missing Glasgow Coma Scale (GCS) data could impact sepsis label assignment. The paper also notes potential issues with missingness and accuracy of clinical data, common in EHR studies.
Future work suggested includes:
- True Prospective Studies: Conducting a real-world prospective study is essential to validate the model's performance.
- Investigating Repeated Alerts and Alert Logic: Further research is needed to optimize alert strategies, including handling repeated alerts and implementing alert lock-out periods.
- Refining Thresholds: Defining an operationally meaningful threshold for alerts is crucial for clinical implementation.
- Exploring Other Settings: Investigating the model's performance in other hospital settings and patient subpopulations (e.g., pediatric, surgical) is warranted.
- Improving Feature Engineering: Further work on automated feature generation from medication data and exploring the impact of comorbidities over longer patient histories could enhance the model.