Research on Multimodal Fusion of Temporal Electronic Medical Records

Moxuan Ma, Muyu Wang, Binyu Gao, Yichen Li, Jun Huang, Hui ChenBioengineering 2024 · DOI

TL;DR

This paper introduces T-MAG, a novel multimodal fusion model for Electronic Medical Records (EMRs) that effectively integrates static and temporal data from clinical notes and tabular records. T-MAG achieves superior performance in predicting in-hospital mortality and long hospital stays by employing multimodal adaptation gates and an attention-backtracking module to capture complex temporal dependencies.

What Problem Does This Paper Address?

Electronic Medical Records (EMRs) are a rich source of patient information, encompassing diverse data types like clinical notes, lab results, and demographic details. However, effectively leveraging this multimodal data, especially the temporal aspects that capture disease progression, remains a significant challenge. Existing EMR research often focuses on single modalities or simpler fusion techniques, failing to fully exploit the intricate temporal dependencies and the interplay between different data types. This gap hinders the development of more accurate and comprehensive predictive models for critical healthcare outcomes.

Methods

The core of the proposed solution is the Time Series Multimodal Adaptation Gate (T-MAG) model. T-MAG processes four distinct data modalities: static tabular, static notes, temporal tabular, and temporal notes.

  • Representation: Each modality is first encoded into a meaningful representation. Static data uses one-hot encoding and doc2vec for tabular and note data, respectively. Temporal data is processed using fully connected networks and LSTMs for tabular data, and Transformer-XL followed by LSTMs for clinical notes, to capture temporal patterns.
  • Fusion: T-MAG employs Multimodal Adaptation Gates (MAGs) for fusion. Crucially, it separates the fusion of static and temporal data to preserve temporal information. MAGs use a primary and auxiliary modality approach, where the auxiliary modality's influence is gated and added as a displacement vector to the primary modality's representation.
  • Attention-Backtracking Module: A key innovation is the attention-backtracking module. This module enhances the model's ability to capture long-term dependencies in temporal data by using a self-attention mechanism to consider information from previous subsequences, complementing the current representation.
  • Final Fusion: The fused static and temporal representations are concatenated and fed into a final LSTM layer to produce the ultimate patient representation for prediction.

Datasets

The study utilized a proprietary dataset from the EMR system of a tertiary hospital in Beijing, China, spanning from January 2014 to December 2016.

  • AMI Dataset: 1271 records of patients diagnosed with Acute Myocardial Infarction (AMI), confirmed via ICD-10 codes I21 and I22.
  • Stroke Dataset: 6450 records of patients diagnosed with stroke, confirmed via ICD-10 codes I60-I64, I66, and I67.8.

The data was preprocessed by padding temporal data to a 30-day interval and segmenting it into 3-day subsequences.

Key Insights and Ideas

  • Clinical Notes are Crucial: The study highlights the significant importance of clinical notes (both static and temporal) for predictive tasks, as selecting them as the primary modality consistently yielded the best results.
  • Temporal Information is Key: The "only static" subset performed the worst, underscoring that neglecting time-series data, especially temporal notes, significantly degrades predictive performance.
  • Preserving Temporal Structure: Separating the fusion of static and temporal data is vital to avoid obscuring the inherent temporal sequence information present in time-series records.
  • Long-Term Dependencies Matter: The attention-backtracking module proved essential for capturing long-term dependencies, leading to a noticeable performance improvement, particularly in tasks like predicting prolonged hospital stays.

Key Results

The T-MAG model demonstrated superior performance across two clinical tasks: predicting in-hospital mortality and predicting a long hospital stay (defined as >14 days).

  • AMI Dataset: T-MAG achieved AUROCs of 0.928 for in-hospital mortality and 0.881 for long hospital stay.
  • Stroke Dataset: T-MAG achieved AUROCs of 0.954 for in-hospital mortality and 0.847 for long hospital stay.

In comparative experiments, T-MAG consistently outperformed other models, including DNNs, LSTMs, and other advanced fusion methods like Crossformer and MISTS-fusion, by a significant margin. For instance, on the AMI dataset for in-hospital mortality, T-MAG's AUROC (0.928) was substantially higher than DNN (0.748) and Fusion-LSTM (0.828). Ablation studies showed that removing the attention-backtracking module led to a performance decline, with AUROCs dropping by approximately 0.039-0.044 on the AMI dataset and 0.02-0.056 on the Stroke dataset.

Limitations and Future Work

  • Limited Sample Size: The study used data from a single hospital, which could lead to overfitting and limit generalizability.
  • Simplistic Temporal Handling: The methods for handling missing data and capturing temporal dependencies in temporal data are considered relatively simplistic and could be further refined.
  • Limited Clinical Tasks: The evaluation was confined to two specific clinical tasks (in-hospital mortality and long hospital stay).

Future work could involve:

  • Expanding the model's applicability to other clinical tasks, such as predicting specific complications or performing multiclass disease diagnosis.
  • Exploring the integration of other modalities, such as clinical imaging.
  • Adapting the model for English-language datasets.
  • Developing more sophisticated techniques for handling temporal data intricacies.

Summary updated

Abstract

The surge in deep learning-driven EMR research has centered on harnessing diverse data forms. Yet, the amalgamation of diverse modalities within time series data remains an underexplored realm. This study probes a multimodal fusion approach, merging temporal and non-temporal clinical notes along with tabular data. We leveraged data from 1271 myocardial infarction and 6450 stroke inpatients at a Beijing tertiary hospital. Our dataset encompassed static, and time series note data, coupled with static and time series table data. The temporal data underwent a preprocessing phase, padding to a 30-day interval, and segmenting into 3-day sub-sequences. These were fed into a long short-term memory (LSTM) network for sub-sequence representation. Multimodal attention gates were implemented for both static and temporal subsequence representations, culminating in fused representations. An attention-backtracking module was introduced for the latter, adept at capturing enduring dependencies in temporal fused representations. The concatenated results were channeled into an LSTM to yield the ultimate fused representation. Initially, two note modalities were designated as primary modes, and subsequently, the proposed fusion model was compared with comparative models including recent models such as Crossformer. The proposed model consistently exhibited superior predictive prowess in both tasks. Removing the attention-backtracking module led to performance decline. The proposed model consistently shows excellent predictive capabilities in both tasks. The proposed method not only effectively integrates data from the four modalities, but also has a good understanding of how to handle irregular time series data and lengthy clinical texts. An effective method is provided, which is expected to be more widely used in multimodal medical data representation.
BibTeX
@inproceedings{ma2024research,
  author = {Moxuan Ma and Muyu Wang and Binyu Gao and Yichen Li and Jun Huang and Hui Chen},
  title = {{Research on Multimodal Fusion of Temporal Electronic Medical Records}},
  booktitle = {Bioengineering},
  year = {2024},
  doi = {10.3390/bioengineering11010094},
}
Figure