Combining LSTM and Latent Topic Modeling for Mortality Prediction

Yohan Jo, Lisa Lee, Shruti PalaskararXiv 2017 · arXiv

Abstract

Predicting patient mortality in Intensive Care Units (ICUs) is a critical task for clinical decision support. This paper introduces an end-to-end neural architecture that jointly learns to predict mortality and extract latent topics from clinical notes. By integrating Long Short-Term Memory (LSTM) networks with LDA-inspired topic modeling layers, the proposed models significantly outperform traditional two-stage baselines. This joint training approach ensures that the discovered topics are optimized for the predictive task, offering a path toward more interpretable clinical models.

Introduction

Mortality prediction in the ICU is essential for prioritizing care and managing resources. However, clinical data presents two major hurdles: temporal complexity and the need for accountability. Patient health is a dynamic trajectory, requiring models that can process sequential clinical notes. Furthermore, clinicians require interpretable rationales—specific clinical themes or "topics"—to validate high-risk predictions.

Figure 1: Temporal setup for mortality prediction using clinical documents collected at 12-hour intervals post-admission.

Traditional methods often utilize Latent Dirichlet Allocation (LDA) to extract topics, which are then used as features for a separate classifier. These "two-stage" processes are suboptimal because topic discovery is not guided by the final prediction task.

Methods

We propose three joint, end-to-end neural network architectures that integrate an LSTM for temporal modeling with specialized topic-modeling layers.

Figure 5: Comparison of the baseline LSTM+LDA model with the proposed joint architectures: LSTM+E, LSTM+E+D, and LSTM+E+T+D.

As shown in Figure 5, the architectures vary in how they process the bag-of-words (BoW) representation ($x^{(t)}$) into a latent topic vector ($z^{(t)}$):

  • LSTM+E (Encoder): Encodes the BoW directly into a latent vector for the LSTM.
  • LSTM+E+D (Encoder-Decoder): Adds a Decoder to reconstruct the original BoW from the latent vector, acting as a regularizer to ensure cohesive topic representations.
  • LSTM+E+T+D (Encoder-Transcoder-Decoder): Introduces a "Transcoder" layer and L1 regularization to mimic the sparsity of traditional LDA.

The models are trained to minimize a joint loss function comprising mortality prediction error and document reconstruction error.

Experimental Setup

The models were evaluated using the MIMIC-III dataset, a large-scale database of de-identified ICU patient records. The study focused on textual clinical notes, excluding discharge summaries to prevent label leakage. The final dataset included 36,218 patients, with text preprocessed into BoW representations using the top 500 tf-idf words.

Key Results

The joint models consistently outperformed the LSTM+LDA baseline across multiple mortality prediction tasks:

  • Predictive Accuracy: The LSTM+E model improved AUC by approximately +4% for in-hospital mortality and +7% for 1-year post-discharge mortality compared to the baseline.
  • Temporal Robustness: LSTM-based models maintained stable performance over longer patient stays, whereas traditional SVM-based models often saw performance degradation.
  • Latent Space Separation: Joint training allowed the model to learn topics specifically predictive of outcomes, resulting in clearer clustering of "died" versus "survived" instances in the latent representation.

Discussion and Limitations

Despite improved predictive performance, interpretability remains a challenge. The learned topics were often noisy, containing medical jargon and typos (e.g., "evenign"). Additionally, aggregating notes into 12-hour windows may have mixed disparate clinical contexts, such as nursing and radiology reports. Future work will focus on more aggressive text cleaning and finer-grained temporal modeling to improve topic quality and clinical utility.

Summary updated

Abstract

There is a great need for technologies that can predict the mortality of patients in intensive care units with both high accuracy and accountability. We present joint end-to-end neural network architectures that combine long short-term memory (LSTM) and a latent topic model to simultaneously train a classifier for mortality prediction and learn latent topics indicative of mortality from textual clinical notes. For topic interpretability, the topic modeling layer has been carefully designed as a single-layer network with constraints inspired by LDA. Experiments on the MIMIC-III dataset show that our models significantly outperform prior models that are based on LDA topics in mortality prediction. However, we achieve limited success with our method for interpreting topics from the trained models by looking at the neural network weights.
BibTeX
@misc{jo2017combining,
  author = {Yohan Jo and Lisa Lee and Shruti Palaskar},
  title = {{Combining LSTM and Latent Topic Modeling for Mortality Prediction}},
  year = {2017},
  eprint = {1709.02842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.02842},
}
Figure