Predicting multiple sclerosis disease severity with multimodal deep neural networks

Kai Zhang, John A. Lincoln, Xiaoqian Jiang, Elmer V. Bernstam, Shayan ShamsarXiv 2023 · OpenReview

TL;DR

This paper introduces a multimodal deep learning framework that integrates Electronic Health Records (EHR), neuroimaging (MRI), and clinical notes to predict Multiple Sclerosis (MS) disease severity. By leveraging longitudinal patient data, their model achieves a significant improvement in prediction accuracy (up to 25% increase in AUROC) compared to single-modality approaches, offering a more comprehensive view of disease progression.

What Problem Does This Paper Address?

Multiple Sclerosis (MS) is a chronic, debilitating neurological disease that damages the brain and spinal cord. Accurately assessing MS severity is crucial for timely intervention and slowing disease progression. The standard clinical measure, the Expanded Disability Status Scale (EDSS), is subjective and requires expert evaluation, limiting its widespread and frequent application. While previous research has explored using machine learning with single data sources (like MRI or clinical notes), these approaches have been hampered by data limitations and model simplicity, leading to suboptimal prediction accuracy. This paper aims to bridge this gap by developing a more robust predictive model that harnesses the power of multiple, diverse data modalities from patient EHRs and neuroimaging.

Methods

The core of this work is a novel multimodal deep neural network designed for data fusion. The architecture employs an encoder-decoder structure with a self-attention mechanism.

  • Encoders: Each data modality is processed by a specialized encoder:
  • Structured EHR: This includes laboratory tests, vital signs, and medication data. A series of 1D convolutional neural networks (CNNs) with attention mechanisms are used to handle the irregular time-series nature of this data, creating a fixed-dimension embedding.
  • Neuroimaging (MRI): ResNet models are employed to extract features from different MRI sequences (T1-pre, T1-post, T2, FLAIR, PD). The goal here is to learn latent vector representations rather than direct classification.
  • Clinical Notes: A graph attention convolution model, leveraging pre-trained medical word embeddings, is used to process free-text clinical notes. This model treats the document as a word co-occurrence network.
  • Multi-modality Fusion: The latent representations from each encoder are concatenated and fed into a bidirectional GRU (Bi-GRU) network. A self-attention layer within the decoder helps weigh the importance of different modalities.
  • Longitudinal Data: Crucially, the model utilizes longitudinal data, incorporating both current and historical patient visits, to capture disease progression over time.
  • Prediction: The final output of the decoder is combined with demographic data and passed through a fully-connected layer to predict EDSS milestones.

Datasets

The study utilizes a dataset comprising 300 MS patients from a local academic medical center. Each patient's data includes:

  • Neuroimaging Data: 360 MRI scans across five sequences (T1-pre, T1-post, T2, PD, FLAIR) from a Philips 3.0T Ingenia scanner.
  • Structured EHR Data: This includes laboratory tests, vital sign observations, medication administrations, and demographic information collected during clinic visits.
  • Clinical Notes: Free-text physician notes detailing patient status and health information.

The dataset is characterized by longitudinal records, with patients having multiple clinic visits. The EDSS score, evaluated by physicians at each visit, serves as the ground truth for prediction.

Key Insights and Ideas

  • Multimodality is Key: The most significant insight is that combining diverse data sources (EHR, MRI, clinical notes) dramatically improves MS severity prediction accuracy compared to using any single modality. This underscores the complex nature of MS and the need for a holistic view.
  • Longitudinal Data Matters: Utilizing historical patient data alongside current information is vital for accurately capturing disease progression and predicting future severity. This moves beyond a static snapshot to a dynamic understanding of the disease.
  • Clinical Notes Pack a Punch: Surprisingly, clinical notes, despite being unstructured text, contribute significantly to prediction accuracy, often outperforming MRI data alone. This suggests that physicians' narrative descriptions implicitly capture crucial information that might be missed by structured data or imaging alone.
  • Attention Mechanisms Enhance Interpretability: The use of attention mechanisms not only boosts prediction accuracy but also provides insights into which features and modalities are most influential in predicting MS severity, aiding in understanding the model's decision-making process.

Key Results

The multimodal deep learning framework demonstrated superior performance in predicting MS disease severity.

  • AUROC Improvement: The proposed multimodal model achieved up to a 25% increase in the Area Under the Receiver Operating Characteristic curve (AUROC) compared to single-modal models.
  • Top Performance: The combination of MRI, clinical notes, and structured EHR data yielded the highest prediction accuracy, with an AUROC of 0.9301.
  • Modality Importance: While all modalities contributed, clinical notes and MRI data were found to be particularly informative. Structured EHR data, while useful, contributed the least compared to the other two.
  • Feature Importance: Analysis of attention weights revealed that specific laboratory tests like "Absolute Neutrophils," "Absolute Lymphocytes," and "Absolute Monocytes" were highly important. Certain medications also showed significant attention weights.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Generalizability of MRI: The study used MRI data from a single scanner, which may limit the generalizability of the model to data from different scanners. Future work should involve collecting data from multiple institutions and scanners to verify the impact of MRI.
  • Defining "Aggressive" MS: The paper notes that MS severity can be a relative concept and varies based on factors like disease duration. Future research could explore predicting MS progression rate and distinguishing between "benign" and "aggressive" disease trajectories, potentially using survival analysis.
  • Data Size: The study was conducted on a cohort of 300 MS patients. Expanding the dataset and evaluating generalizability across different institutions is a crucial next step.
  • Cost-Effectiveness: Future work could investigate which specific EHR data points are most cost-effective to collect for maximizing prediction accuracy, guiding future data collection strategies.

Summary updated

Abstract

Multiple Sclerosis (MS) is a chronic disease developed in human brain and spinal cord, which can cause permanent damage or deterioration of the nerves. The severity of MS disease is monitored by the Expanded Disability Status Scale (EDSS), composed of several functional sub-scores. Early and accurate classification of MS disease severity is critical for slowing down or preventing disease progression via applying early therapeutic intervention strategies. Recent advances in deep learning and the wide use of Electronic Health Records (EHR) creates opportunities to apply data-driven and predictive modeling tools for this goal. Previous studies focusing on using single-modal machine learning and deep learning algorithms were limited in terms of prediction accuracy due to the data insufficiency or model simplicity. In this paper, we proposed an idea of using patients' multimodal longitudinal and longitudinal EHR data to predict multiple sclerosis disease severity at the hospital visit. This work has two important contributions. First, we describe a pilot effort to leverage structured EHR data, neuroimaging data and clinical notes to build a multi-modal deep learning framework to predict patient's MS disease severity. The proposed pipeline demonstrates up to 25% increase in terms of the area under the Area Under the Receiver Operating Characteristic curve (AUROC) compared to models using single-modal data. Second, the study also provides insights regarding the amount useful signal embedded in each data modality with respect to MS disease prediction, which may improve data collection processes.
BibTeX
@misc{zhang2023predicting,
  author = {Kai Zhang and John A. Lincoln and Xiaoqian Jiang and Elmer V. Bernstam and Shayan Shams},
  title = {{Predicting multiple sclerosis disease severity with multimodal deep neural networks}},
  year = {2023},
  eprint = {2304.04062},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.04062},
}
Figure