Developing a deep learning system to drive the work of the critical care outreach team

Georgina Kennedy, John Rihari‐Thomas, Mark Dras, Blanca GallegomedRxiv 2020 · DOI

Abstract

Care of patients at risk of deterioration on acute medical and surgical wards requires timely identification, increased monitoring and robust escalation procedures. The critical care outreach role brings specialist-trained critical care nurses and physicians into acute wards to facilitate these processes. Performing this role is challenging, as the breadth of information synthesis required is both high and rapidly updating. We propose a novel automated ‘watch-list’ to identify patients at high risk of deterioration, to help prioritise the work of the outreach team. This system takes data from the electronic medical record in real-time and creates a discrete tokenized trajectory, which is fed into a recurrent neural network model. These models achieve an AUROC of 0.928 for inpatient death and 0.778 for unplanned ICU admission (within 24 hours), which compares favourably with existing early warning scores and is comparable with proof of concept deep learning systems requiring significantly more input data. Based on these results, we can conclude that it is technically feasible to build a set of predictive models that meet the needs of the critical care outreach role, based on a limited set of real-time clinical data.

Introduction

In acute hospital settings, identifying patients whose health is about to deteriorate is a critical task. Critical Care Outreach Teams (CCOTs) are tasked with identifying these patients before a crisis occurs. However, CCOTs are often overwhelmed by the volume of data. Traditional tools like the National Early Warning Score (NEWS) rely heavily on vital signs. In many clinical environments, these vitals are recorded with significant delays, limiting their utility for real-time automated alerts. There is a clear need for systems that can synthesize diverse administrative and clinical data—such as lab results, medication orders, and ward movements—to prioritize patient care.

Methods

The proposed deep learning architecture treats a patient’s hospital journey as a sequence of events, analogous to words in a sentence.

  1. Tokenization and Trajectory Construction: Every Electronic Medical Record (EMR) entry, including diagnoses, pathology results, medications, and ward transfers, is converted into a discrete token. Continuous data, such as laboratory values, are "deciled" to fit this format. This creates a discrete tokenized trajectory for each patient.
Figure 1: Example of a patient trajectory showing historical admissions, emergency department (ED) stay, and prediction targets.
  1. Time-Sensitive Embeddings: To capture the temporal importance of events, a modified skip-gram algorithm was used to create vector representations (embeddings) where the relationship weight between events decays based on elapsed time.
  2. Model Architecture: The system employs three sub-models trained jointly:
  3. Model 1: A feed-forward network for "flat" features (e.g., age, insurance status).
  4. Model 2: A Bidirectional Long Short-Term Memory (LSTM) network processing the most recent 500 tokens in the trajectory.
  5. Model 3: A final layer concatenating the outputs of the first two models to produce risk scores for multiple time horizons.
Figure 2: The three-part model architecture combining flat features and recurrent trajectory features.

Datasets

The study utilized a retrospective dataset from a metropolitan quaternary-care hospital in Sydney, Australia, spanning 2008–2016. The dataset included 192,883 hospitalizations across 92,802 adult patients, encompassing nearly 900,000 diagnoses and over 12 million medication events.

Results

The model demonstrated high predictive accuracy for events within 24 hours:

  • Inpatient Mortality: Achieved an AUROC of 0.928, outperforming the reported baseline for NEWS (0.89).
  • Unplanned ICU Admission: Achieved an AUROC of 0.778.

Beyond raw metrics, the model's clinical utility was assessed using a "72-hour window of tolerance." The system showed a Workup to Detection Ratio (WDR) of 1 in 12 for ICU admissions, suggesting a manageable false alarm rate for clinical teams.

Discussion and Conclusion

A key insight from this work is that models can outperform traditional vital-sign-based scores by utilizing "proxy" data. Clinical actions, such as ordering specific high-intensity medications or lab tests, serve as strong signals of patient risk. By treating clinical history as a "language," the model learns the context of a patient's stay.

While the model shows promise, challenges remain regarding calibration and generalizability across different hospital systems with varying coding practices. Future work will focus on real-world pilot testing to evaluate the impact of this automated "watch-list" on patient outcomes. In conclusion, it is technically feasible to build predictive models that meet the needs of critical care outreach using a limited set of real-time clinical data.

Summary updated

Abstract

Abstract Background Care of patients at risk of deterioration on acute medical and surgical wards requires timely identification, increased monitoring and robust escalation procedures. The critical care outreach role brings specialist-trained critical care nurses and physicians into acute wards to facilitate these processes. Performing this role is challenging, as the breadth of information synthesis required is both high and rapidly updating. We propose a novel automated ‘watch-list’ to identify patients at high risk of deterioration, to help prioritise the work of the outreach team. Results This system takes data from the electronic medical record in real-time and creates a discrete tokenized trajectory, which is fed into a recurrent neural network model. These models achieve an AUROC of 0.928 for inpatient death and 0.778 for unplanned ICU admission (within 24 hours), which compares favourably with existing early warning scores and is comparable with proof of concept deep learning systems requiring significantly more input data. Conclusions Based on these results, we can conclude that it is technically feasible to build a set of predictive models that meet the needs of the critical care outreach role, based on a limited set of real-time clinical data.
BibTeX
@misc{kennedy2020developing,
  author = {Georgina Kennedy and John Rihari‐Thomas and Mark Dras and Blanca Gallego},
  title = {{Developing a deep learning system to drive the work of the critical care outreach team}},
  year = {2020},
  doi = {10.1101/2020.07.07.20148064},
  howpublished = {medRxiv},
}
Figure