Deep Learning to Attend to Risk in ICU

Phuoc Nguyen, Truyen Tran, Svetha Venkatesh

Introduction

Predicting patient outcomes in the Intensive Care Unit (ICU) is notoriously difficult due to the "messy" nature of clinical data. Unlike standard laboratory datasets, ICU physiological time-series (e.g., heart rate, glucose, pH) are irregularly sampled. This "missing data" is often informative missingness: the absence of a measurement typically implies that a clinician deemed the patient stable in that regard. Traditional machine learning models often struggle with these irregular gaps or rely on crude imputation methods that ignore the underlying clinical intent. This paper introduces a deep learning architecture designed to "attend" to the most critical signals and time periods while naturally handling these irregularities.

Methods

The authors propose a hierarchical approach based on Long Short-Term Memory (LSTM) networks, enhanced with two specific levels of attention:

  1. Sensing Layer (Input Gating): The model uses the LSTM’s internal input gates to decide which parts of the current physiological measurements are informative. If a new measurement is redundant, the gate "shuts," and the model relies on its internal memory of the patient's state.
  2. Reasoning Layer (Temporal Attention): Multiple "reading heads" look across the entire 48-hour history and assign weights to different time intervals. This allows the model to aggregate "illness states" into a single feature vector $z$, focusing on the moments where the patient was most at risk.
  3. Bidirectional Processing: A Bidirectional LSTM (BiLSTM) processes the time-series both forward and backward to capture a comprehensive context of the patient's trajectory.
Hierarchical attention architecture for ICU risk prediction.

Data is divided into 3-hour windows. Within each window, five statistics (min, max, mean, median, and standard deviation) are extracted for each biomarker to create a fixed-length input vector for the LSTM.

Key Results

The model was evaluated using the PhysioNet Challenge 2012 dataset, consisting of 4,000 ICU patients. The task was binary classification of hospital mortality. The proposed BiLSTM with attention outperformed all baselines in terms of Area Under the ROC Curve (AUC):

Method AUC
Logistic Regression (48-hr stats) 0.791
GRU-mean (Standard Imputation) 0.820
LSTM (Mean Pooling) 0.825
LSTM (With Attention) 0.833
BiLSTM (With Attention) 0.839

The results demonstrate that modeling temporal dynamics with LSTMs is superior to static statistics, and adding attention mechanisms provides a significant performance boost over simple averaging (mean pooling) of states.

Key Insights and Interpretability

A core contribution of this work is the use of Attention as an Interpretability Tool. By visualizing the attention weights (the "reading heads"), clinicians can see exactly when the model detected a spike in risk. This transforms the "black box" of deep learning into a tool that can provide "alerts" during specific critical windows.

Visualization of attention weights and physiological signals for two patients.

Furthermore, the LSTM's gating mechanism acts as a dynamic imputation tool, maintaining the previous "illness state" when data is missing. The use of multiple attention heads allows the model to monitor different types of clinical risks simultaneously, such as respiratory distress and sepsis.

Limitations and Future Work

While the model handles missingness well through gating, the initial preprocessing still relies on simple mean imputation within 3-hour windows. Integrating more sophisticated imputation (like GRU-D) could further improve results. Additionally, future work could explore multi-resolution modeling to capture both slow-moving trends and sudden acute events, as some ICU events occur much faster than the 3-hour window allows. Finally, explicitly incorporating measurement uncertainty into the reasoning process is a logical next step for clinical deployment.

Summary updated

BibTeX
@inproceedings{nguyen2017deep,
  author = {Phuoc Nguyen and Truyen Tran and Svetha Venkatesh},
  title = {{Deep Learning to Attend to Risk in ICU}},
  booktitle = {KHD @ IJCAI},
  year = {2017},
  url = {https://openreview.net/forum?id=S14Mq4zd-S},
}
Figure