Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

Kwanhyung Lee, Soojeong Lee, Sangchul Hahn, Heejung Hyun, Edward Choi, Byungeun Ahn, Joohyung Lee

TL;DR

This paper introduces a novel framework for learning from Electronic Health Records (EHRs) that are incomplete due to missing data modalities and irregular sampling. Their approach, UMSE (Unified Multi-modal Set Embedding) and MAA (Modality-Aware Attention), effectively handles these challenges, leading to improved predictions for critical patient outcomes like mortality and the need for interventions.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of patient information, encompassing various data types like time-series vital signs, medical images, and clinical notes. However, effectively leveraging this multi-modal data for clinical prediction is hindered by two major issues:

  1. Missing Modalities: Not all data types are available for every patient. For instance, a patient might have vital signs but lack an X-ray or detailed clinical notes. This makes it difficult to use standard multi-modal models that expect complete data.
  2. Irregular and Asynchronous Sampling: Data points within a modality (especially time-series data) are often not collected at fixed intervals. This temporal irregularity complicates how information is processed and how relationships between different data types are modeled.

Existing methods often struggle with these problems. Some rely on error-prone imputation techniques or discard temporal information from non-time-series data. This paper aims to bridge this gap by developing a unified approach that can handle both missing data and temporal complexities across diverse EHR modalities.

Methods

The core of the proposed solution lies in two key components:

  • Unified Multi-modal Set Embedding (UMSE): This novel embedding technique treats all EHR modalities uniformly, regardless of their type (time-series, image, text). It addresses irregular and asynchronous sampling by encoding each observation as a triplet of (value, time, feature type). Crucially, UMSE shares a time embedding function across all modalities, enabling the model to capture temporal relationships between them. This avoids the need for separate imputation modules or discarding time information from non-time-series data.
  • Modality-Aware Attention (MAA) with Skip Bottleneck (SB): This module is designed to handle missing modalities during the fusion process.
  • Skip Bottleneck (SB): This component enhances the Multimodal Bottleneck Transformer (MBT) architecture to gracefully handle missing modalities. It achieves this by feeding random numbers (e.g., zero-vectors) for missing modalities and selectively skipping bottleneck tokens from missing modalities during fusion.
  • Modality-Aware Attention (MAA): Instead of simply averaging information from different modalities (as in standard attention mechanisms), MAA assigns distinct attention scores to each modality. This allows the model to prioritize more informative modalities for specific prediction tasks, leading to better performance. The paper explores three variants: Time-Series Attention (TSA), Clinical-Task-Aware Attention (CTAA), and Average Attention (AA).

Datasets

The study utilizes three publicly available EHR datasets:

  • MIMIC-IV: A comprehensive dataset containing ICU patient data, including vital signs, lab results, and demographics.
  • MIMIC-CXR: A large dataset of chest X-ray images from ICU patients.
  • MIMIC-ED: Contains chief complaint text data from Emergency Department visits.

These datasets were merged to create a unified multi-modal dataset for patient-level analysis.

Key Insights and Ideas

  • Unified Embedding is Key: The UMSE approach demonstrates that a single, unified embedding strategy can effectively capture temporal dynamics across diverse EHR modalities, a significant improvement over methods that treat modalities separately or discard temporal information.
  • Modality Importance Varies: The Modality-Aware Attention (MAA) highlights that not all data sources are equally important for every clinical prediction task. For example, mortality prediction seems to rely more heavily on time-series data, while other tasks might benefit more from image or text data.
  • Handling Missing Data is Crucial: The proposed Skip Bottleneck (SB) mechanism effectively allows existing powerful architectures like MBT to work with incomplete data, a common scenario in real-world healthcare.
  • Time Matters for All Modalities: The study emphasizes that providing occurrence time information for non-time-series data (like images and text) consistently improves performance, a factor often overlooked in prior EHR research.

Key Results

The proposed UMSE and MAA with SB model achieved state-of-the-art performance across three critical clinical prediction tasks:

  • Mortality Prediction: Achieved the highest AUPRC/AUROC scores (e.g., 0.864/0.970 for trimodal fusion with CTAA).
  • Vasopressor Need Prediction: Also demonstrated superior performance (e.g., 0.837/0.928 for trimodal fusion with CTAA).
  • Intubation Need Prediction: Showed the best results (e.g., 0.868/0.934 for trimodal fusion with CTAA).

The study also found that:

  • Losing EHR Image data negatively impacts predictive performance more than losing EHR Text data.
  • The effectiveness of different strategies for handling missing modalities (like Missing-Modality Augmentation) varies depending on the clinical task.

Limitations and Future Work

The authors acknowledge several limitations:

  • No Single Solution for Missing Modality: Despite their advancements, they found that no single approach could universally improve robustness against missing modalities across all tasks and data types. Specifically, improving robustness for missing EHR text in mortality and intubation need prediction remains a challenge.
  • Focus on Specific Modalities: The current work focuses on time-series, image, and text data. Future work could explore incorporating other EHR modalities.

The authors suggest that further research into missing modality handling is a promising direction. They also highlight that their UMSE method is advantageous not only for EHR time-series but also for other modalities, suggesting its broad applicability.

Summary updated

Abstract

Electronic Health Record (EHR) provides abundant information through various modalities. However, learning multi-modal EHR is currently facing two major challenges, namely, 1) irregular and asynchronous sampling and 2) modality missing. Moreover, a lack of shared embedding function across modalities can discard the temporal relationship between different EHR modalities. On the other hand, most EHR studies are limited to relying only on EHR Times-series, and therefore, missing modality in EHR has not been well-explored. Therefore, in this study, we introduce a Unified Multi-modal Set Embedding (UMSE) and Modality-Aware Attention (MAA) with Skip Bottleneck (SB). UMSE treats all EHR modalities without a separate imputation module or error-prone carry-forward, whereas MAA with SB learns missing modal EHR with effective modality-aware attention. Our model outperforms other baseline models in mortality, vasopressor need, and intubation need prediction with the MIMIC-IV dataset.
BibTeX
@inproceedings{lee2023learning,
  author = {Kwanhyung Lee and Soojeong Lee and Sangchul Hahn and Heejung Hyun and Edward Choi and Byungeun Ahn and Joohyung Lee},
  title = {{Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention}},
  booktitle = {MLHC},
  year = {2023},
  url = {https://openreview.net/forum?id=alRbQrfMkO},
}
Figure