Towards unstructured mortality prediction with free-text clinical notes

Mohammad Hashir, Rapinder Sawhney

TL;DR

This paper introduces a hierarchical deep learning model (HCR) that leverages unstructured clinical notes to predict in-hospital mortality. By treating each note individually and then modeling their temporal relationships, the HCR achieves superior prediction performance compared to traditional severity scores and models relying solely on structured clinical time series, demonstrating the significant untapped potential of free-text medical data.

What Problem Does This Paper Address?

Predicting patient mortality in intensive care units (ICUs) is crucial for quality assessment, clinical decision-making, and administrative purposes. While established severity scoring systems (like APACHE and SAPS) are widely used, they have limitations: they can suffer from ambiguous interpretation, bias, loss of temporal information, and poor generalizability across different settings. Electronic Health Records (EHRs) contain a wealth of unstructured free-text clinical notes, which hold rich, subjective information about a patient's condition. However, this valuable data is often underutilized in mortality prediction models because traditional machine learning approaches struggle with raw text. This paper aims to bridge this gap by developing a method that effectively extracts predictive signals from these unstructured notes, potentially leading to more accurate and personalized mortality predictions.

Methods

The core of this work is a Hierarchical Convolutional Neural Network (HCR) designed to process multiple clinical notes from a single patient's hospital stay. The HCR has two main components:

  • Semantical Module: This module uses convolutional neural networks (CNNs) to capture semantic information within individual clinical notes. It processes each note independently, converting it into a "document vector." This approach automates feature extraction from the text, a task that previously required manual NLP expertise and was prone to bias.
  • Temporal Module: This module takes the document vectors generated by the semantical module and processes them sequentially using recurrent neural networks (RNNs), specifically Gated Recurrent Units (GRUs). This allows the model to capture the evolution of a patient's condition over time, modeling the temporal relationships between different notes.

The model is designed to be "dynamic," meaning it uses only data available up to a specific point in the patient's stay for prediction, mimicking real-world clinical scenarios.

Datasets

The study utilizes the MIMIC-III (Medical Information Mart for Intensive Care III) dataset. This is a publicly available, de-identified ICU database containing comprehensive clinical data from nearly 60,000 critical care admissions. The dataset includes vital signs, medications, lab results, and importantly for this study, a large volume of unstructured clinical notes. The authors created two corpora from MIMIC-III: one for pre-training embeddings and another for training the prediction model.

Key Insights and Ideas

  • Unstructured Notes Hold Significant Predictive Power: The study strongly suggests that free-text clinical notes contain valuable information for mortality prediction that is often missed by structured data or traditional scoring systems.
  • Hierarchical Approach Captures Both Content and Context: The HCR's ability to first understand the meaning within each note (semantical module) and then model how these meanings change over time (temporal module) is key to its success.
  • Less Preprocessing, More Performance: The HCR achieved higher metrics than structured data approaches despite requiring significantly less cleaning and preprocessing, highlighting the efficiency of deep learning in handling raw text.
  • "Dynamic" Prediction is Feasible and Effective: The model's ability to predict mortality based on data available up to a certain point in time is crucial for clinical utility and was successfully demonstrated.

Key Results

The HCR models, particularly the multi-modal version (MM-HCR) that combines notes with structured clinical time series, outperformed traditional severity scores (APS-III, OASIS, SAPS-II) across various metrics (AUROC and AUPRC) and time windows (12, 24, and 48 hours).

  • For the W=24 experiment, the MM-HCR achieved an AUROC of 0.8787 and an AUPRC of 0.3327, significantly outperforming the best severity score (SAPS-II: AUROC 0.8048, AUPRC 0.2356).
  • Even the Notes-HCR (using only notes) performed better than severity scores, achieving an AUROC of 0.8675 and AUPRC of 0.3010 for W=24.
  • The gap between note-based models and clinical time series models narrowed as the time window increased (from 12 to 48 hours), but notes consistently provided higher AUPRC.

Limitations and Future Work

  • Interpretability: A major limitation acknowledged is the lack of interpretability of the HCR model. It's difficult to pinpoint exactly which parts of the notes contribute to the mortality prediction, making it a "black-box" system.
  • Future Directions:
    • Improved Temporal Modeling: Exploring more advanced temporal architectures like transformers could further enhance the model's ability to capture temporality.
  • Incorporating Metadata: Utilizing metadata associated with notes (e.g., note type, author) could enrich the model's understanding.
  • Enhanced Interpretability: Developing methods to make the HCR more interpretable is a key area for future research, potentially by adapting techniques like DeepLIFT.
  • Multi-modal Fusion: Further exploring the fusion of notes with other data modalities (e.g., medical images, static demographic data) could lead to even more robust predictions.
  • Transfer Learning: The authors suggest that pre-training HCR models on large public datasets like MIMIC-III and then fine-tuning them on smaller, private datasets could be highly beneficial for hospitals with limited data.

Summary updated

Abstract

Healthcare data continues to flourish yet a relatively small portion, mostly structured, is being utilized effectively for predicting clinical outcomes. The rich subjective information available in unstructured clinical notes can possibly facilitate higher discrimination but tends to be under-utilized in mortality prediction. This work attempts to assess the gain in performance when multiple notes that have been minimally preprocessed are used as an input for prediction. A hierarchical architecture consisting of both convolutional and recurrent layers is used to concurrently model the different notes compiled in an individual hospital stay. This approach is evaluated on predicting in-hospital mortality on the MIMIC-III dataset. On comparison to approaches utilizing structured data, it achieved higher metrics despite requiring less cleaning and preprocessing. This demonstrates the potential of unstructured data in enhancing mortality prediction and signifies the need to incorporate more raw unstructured data into current clinical prediction methods.
BibTeX
@article{hashir2020towards,
  author = {Mohammad Hashir and Rapinder Sawhney},
  title = {{Towards unstructured mortality prediction with free-text clinical notes}},
  journal = {J. Biomed. Informatics},
  year = {2020},
  url = {https://openreview.net/forum?id=Ftxdbe2c2FA},
}
Figure