Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction

Eunji Jun, Ahmad Wisnu Mulyadi, Jaehun Choi, Heung-Il Suk

TL;DR

This paper introduces a novel deep learning architecture that addresses the "messiness" of Electronic Health Records (EHR) by explicitly modeling the uncertainty of missing data. By using a Variational Recurrent Neural Network (VRNN) coupled with a specialized "Uncertainty-Gated" GRU cell, the model learns to "mute" unreliable imputed values, achieving state-of-the-art results in predicting hospital mortality.

What Problem Does This Paper Address?

Electronic Health Records (EHR) are notoriously difficult to model. They are "sparse" (data is missing because a test wasn't ordered), "noisy" (sensor errors), and "non-stationary" (a patient's condition changes over time).

The standard approach is imputation: filling in the blanks with a "best guess" based on previous values or population means. However, most models treat these guesses as absolute facts. If a model fills a missing blood pressure value with an incorrect guess, that error propagates through the network, leading to poor clinical predictions. The authors argue that a model should not only guess the missing value but also estimate its fidelity (how confident it is in that guess) and use that confidence to weight the information during prediction.

Methods

The authors propose an end-to-end framework that combines three main components:

  1. Stochastic Imputation (VRNN): Instead of predicting a single value for a missing variable, the model predicts a probability distribution (mean and variance). The variance serves as a direct proxy for uncertainty. This is built on a Variational Recurrent Neural Network (VRNN) architecture.
  2. Uncertainty-Gated Attention (GRU-U): This is the core innovation. The authors modified the standard Gated Recurrent Unit (GRU) to create the GRU-U. It introduces an attention mechanism ($\alpha_t$) that uses the calculated uncertainty to gate the input. If the uncertainty for a specific variable is high, the gate reduces its influence on the hidden state update.
  3. Joint Learning: The model is trained using a composite loss function that balances four goals: reconstruction of the data (VRNN loss), consistency between forward and backward passes, imputation accuracy (MAE), and the final classification (mortality prediction). To handle the highly imbalanced nature of mortality data (few deaths vs. many survivors), they utilize Focal Loss.

Datasets

The model was evaluated on two benchmark clinical datasets:

  • MIMIC-III: A large, de-identified dataset of ICU stays. The authors used a subset of ~14,000 patients, focusing on the first 48 hours of data. The missingness rate here is a staggering 93.92%.
  • PhysioNet Challenge 2012: 4,000 ICU records with a missingness rate of approximately 80.51%.

To test imputation robustness, the authors also created scenarios where an additional 5% or 10% of the observed data was intentionally masked.

Key Insights and Ideas

  • "Knowing what you don't know": The model demonstrates a high correlation between its predicted uncertainty and the actual Mean Absolute Error (MAE) of its imputations. This means the model successfully identifies which "guesses" are likely to be wrong.
  • Fidelity-Aware Recurrence: By injecting uncertainty directly into the GRU gates, the model performs "uncertainty propagation." It essentially learns to trust observed data fully and trust imputed data only as much as the variance allows.
  • Task-Specific Imputation: While some models (like BRITS) are slightly better at raw imputation accuracy, this model performs better at the downstream task (mortality prediction). This suggests that perfectly reconstructing a missing value is less important than understanding the reliability of that value for clinical decision-making.

Key Results

The proposed model outperformed several state-of-the-art (SOTA) baselines, including GRU-D, M-RNN, and BRITS.

  • Mortality Prediction (MIMIC-III): Achieved an AUC of 0.865 and an AUPRC of 0.416, surpassing BRITS (0.863 / 0.414).
  • Mortality Prediction (PhysioNet): Achieved an AUC of 0.832, notably higher than the next best SOTA, BRITS (0.824).
  • Ablation Success: Removing the GRU-U cell and using a vanilla GRU caused a significant drop in performance (AUC dropped from 0.832 to 0.794 on PhysioNet), proving that the uncertainty-gating mechanism is the primary driver of the model's success.

Limitations and Future Work

  • Imputation MAE: The model's raw Mean Absolute Error for imputation is higher than BRITS. While this doesn't hurt mortality prediction, it suggests the generative part of the model could be further refined if the goal is pure data reconstruction.
  • Computational Complexity: The stochastic nature of VRNNs and the composite loss function make the model more complex to train and tune compared to deterministic RNNs.
  • Future Directions: The authors suggest applying this uncertainty-aware approach to other clinical tasks, such as predicting the length of stay or identifying specific phenotypes in patients. There is also potential to explore how these uncertainty estimates could be presented to clinicians as a "confidence score" for AI-assisted diagnosis.

Summary updated

Abstract

Electronic health records (EHRs) are characterized as nonstationary, heterogeneous, noisy, and sparse data; therefore, it is challenging to learn the regularities or patterns inherent within them. In particular, sparseness caused mostly by many missing values has attracted the attention of researchers who have attempted to find a better use of all available samples for determining the solution of a primary target task through defining a secondary imputation problem. Methodologically, existing methods, either deterministic or stochastic, have applied different assumptions to impute missing values. However, once the missing values are imputed, most existing methods do not consider the fidelity or confidence of the imputed values in the modeling of downstream tasks. Undoubtedly, an erroneous or improper imputation of missing variables can cause difficulties in the modeling as well as a degraded performance. In this study, we present a novel variational recurrent network that: 1) estimates the distribution of missing variables (e.g., the mean and variance) allowing to represent uncertainty in the imputed values; 2) updates hidden states by explicitly applying fidelity based on a variance of the imputed values during a recurrence (i.e., uncertainty propagation over time); and 3) predicts the possibility of in-hospital mortality. It is noteworthy that our model can conduct these procedures in a single stream and learn all network parameters jointly in an end-to-end manner. We validated the effectiveness of our method using the public data sets of MIMIC-III and PhysioNet challenge 2012 by comparing with and outperforming other state-of-the-art methods for mortality prediction considered in our experiments. In addition, we identified the behavior of the model that well represented the uncertainties for the imputed estimates, which showed a high correlation between the uncertainties and mean absolute error (MAE) scores for imputation.
BibTeX
@article{jun2021uncertainty,
  author = {Eunji Jun and Ahmad Wisnu Mulyadi and Jaehun Choi and Heung-Il Suk},
  title = {{Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  url = {https://openreview.net/forum?id=JJrhukNobUq},
}

Loading related content...

Figure