Towards dynamic multi-modal phenotyping using chest radiographs and physiological data

Nasir Hayat, Krzysztof J. Geras, Farah E. ShamoutarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel "dynamic training" approach for medical patient phenotyping that effectively integrates chest X-rays (CXRs) with physiological data. By allowing modalities to be used flexibly during training, the method significantly improves diagnostic accuracy, demonstrating the power of multi-modal learning in healthcare.

What Problem Does This Paper Address?

Healthcare data is inherently multi-modal, with information coming from various sources like medical images, lab results, and vital signs. While deep learning has shown promise in analyzing single modalities for diagnostics, integrating these diverse data streams remains a challenge. Existing multi-modal approaches often require complete, paired data for all modalities, limiting their applicability when data is missing (a common scenario in real-world clinical settings). This paper tackles the problem of effectively leveraging heterogeneous medical data, even when some modalities are absent, to improve patient phenotyping.

Methods

The core innovation is a "dynamic training" approach. Instead of requiring all data modalities to be present for every training sample, this method allows for flexible integration. During training, the model first learns representations from individual modalities (e.g., CXRs and physiological data) and updates their respective encoders and classifiers using modality-specific losses. Crucially, it also incorporates a step where paired modalities are concatenated and fed into a unified classifier. This dynamic process allows the model to learn from both complete and incomplete data, making it robust to missing modalities. At inference time, the system can fall back to using the available modality if the other is missing.

Datasets

  • MIMIC-IV: Used for physiological data (electronic health records - EHR).
  • MIMIC-CXR: Used for chest radiographs (CXRs).

The authors created a dataset of "paired modalities" by matching ICU stays with corresponding CXRs based on patient identifiers and timestamps. This dataset is used for evaluating the performance of their proposed approach.

Key Insights and Ideas

  • Flexibility is Key: The "dynamic training" approach is a significant departure from rigid multi-modal fusion methods, allowing for more practical application in real-world scenarios where data is often incomplete.
  • Auxiliary Modalities Matter: Even seemingly unrelated auxiliary data, like CXRs for a task primarily driven by physiological data, can provide crucial complementary information that boosts diagnostic performance.
  • Chronic Diseases Benefit Greatly: While acute conditions are often easier to predict, the integration of CXRs showed a particularly strong improvement in the prediction of chronic diseases, suggesting that imaging captures subtle, long-term health patterns.

Key Results

The proposed dynamic multi-modal approach achieved a significant improvement in patient phenotyping accuracy. On a test set where both CXR and EHR data were available, the unified approach yielded an AUROC of 0.764, outperforming the baseline using only physiological data (0.740 AUROC). For a specific set of recurring or chronic diseases, the AUROC improved from 0.747 to 0.798, highlighting the value of the CXR modality.

Limitations and Future Work

  • Limited Modalities: The current work focuses on only two modalities (CXRs and 17 physiological features). Future work could explore integrating more diverse data sources.
  • Data Availability: The approach's reliance on specific datasets (MIMIC-IV and MIMIC-CXR) might limit its direct applicability in low-resource settings where such comprehensive data is unavailable.
  • Phenotype Specificity: The study focused on a specific set of 25 phenotype labels. Expanding to a wider range of conditions, including rare diseases, would increase the work's relevance.
  • Generalizability: Further research is needed to assess the generalizability of this dynamic approach to other medical tasks and different combinations of modalities.

Summary updated

Abstract

The healthcare domain is characterized by heterogeneous data modalities, such as imaging and physiological data. In practice, the variety of medical data assists clinicians in decision-making. However, most of the current state-of-the-art deep learning models solely rely upon carefully curated data of a single modality. In this paper, we propose a dynamic training approach to learn modality-specific data representations and to integrate auxiliary features, instead of solely relying on a single modality. Our preliminary experiments results for a patient phenotyping task using physiological data in MIMIC-IV & chest radiographs in the MIMIC- CXR dataset show that our proposed approach achieves the highest area under the receiver operating characteristic curve (AUROC) (0.764 AUROC) compared to the performance of the benchmark method in previous work, which only used physiological data (0.740 AUROC). For a set of five recurring or chronic diseases with periodic acute episodes, including cardiac dysrhythmia, conduction disorders, and congestive heart failure, the AUROC improves from 0.747 to 0.798. This illustrates the benefit of leveraging the chest imaging modality in the phenotyping task and highlights the potential of multi-modal learning in medical applications.
BibTeX
@misc{hayat2021towards,
  author = {Nasir Hayat and Krzysztof J. Geras and Farah E. Shamout},
  title = {{Towards dynamic multi-modal phenotyping using chest radiographs and physiological data}},
  year = {2021},
  eprint = {2111.02710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02710},
}
Figure