PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo, Zhipeng Wang, Ruiquan Ge, Jianxun Yu, Feiwei Qin, Yuan Tian, Yuqing Peng, Yonghong Li, Changmiao Wang

TL;DR

PE-MVCNet is a novel multimodal deep learning framework designed to predict Pulmonary Embolism (PE) by integrating 3D CT scans with Electronic Medical Record (EMR) data. By leveraging multi-view attention and a specialized cross-modal fusion module, the network achieves a high AUROC of 94.1%, significantly outperforming models that rely on medical imaging alone.

What Problem Does This Paper Address?

Pulmonary Embolism (PE)—a life-threatening blockage in the pulmonary artery—requires rapid and accurate diagnosis to reduce mortality. While Computed Tomography Pulmonary Angiography (CTPA) is the gold standard for diagnosis, interpreting these scans is challenging. A single patient's exam can contain hundreds of slices, and variations in imaging technology can lead to missed diagnoses.

In a clinical setting, doctors don't look at images in a vacuum; they use a patient’s medical history, lab results, and vitals (EMR data) to provide context. However, most existing AI models focus solely on the images. The authors address this gap by creating a system that "thinks" more like a clinician, fusing visual evidence from CT scans with the structured clinical data found in medical records.

Methods

The researchers proposed PE-MVCNet, which consists of three primary components:

  1. Image-only Module: This uses a 3D ResNet backbone enhanced by a Multi-View Coupled Self-Attention (MVCS) block. Unlike standard convolutions that have a limited receptive field, this block uses spatial and "dimensional" attention to capture dependencies across the entire volume of the CT scan, effectively looking for correlations between different slices.
  2. EMR-only Module: This module processes tabular data (like vitals and ICD codes). It uses TabNet to create embeddings and a Multi-Layer Perceptron (MLP) to extract clinical features.
  3. Cross-modal Attention Fusion (CMAF) Module: This is the core innovation. Instead of simply "stacking" the image and EMR features, the CMAF module uses an attention mechanism to align and fuse them. It calculates the "degree of match" between visual features and clinical text, allowing the model to weigh the importance of different data points based on their mutual relevance.
Fig. 1. The overall framework of the proposed PE-MVCNet model for PE prediction. The model comprises the Image-only module, EMR-only module, and Cross-modal Attention Fusion (CMAF) module. The Image-only model employs spatial and dimensional attention to investigate dependency relationships on spatial, channel, and dimensional aspects, respectively. Conversely, the CMAF module is designed to capture the correlation between image and tabular features.

Datasets

The model was trained and evaluated on a publicly accessible dataset from Stanford University Medical Center.

  • Scope: 1,837 axial CTPA exams from 1,794 patients (spanning 2000–2016).
  • Data Types: 3D CT images (1.25mm slice thickness) and corresponding EMR tables (demographics, vitals, inpatient medications, ICD codes, and lab results).
  • Ground Truth: Labels were manually generated by board-certified radiologists.

Key Insights and Ideas

  • The Power of Dimensional Attention: PE isn't confined to a single CT slice. By adding "Dimensional Attention" (as seen in Figure 2), the model can "remember" information from distant slices, which is crucial for identifying emboli that span multiple levels of the pulmonary artery.
  • EMR is a Strong Signal: Interestingly, the EMR-only model performed remarkably well on its own (90.2% AUROC), suggesting that clinical data contains highly predictive markers for PE that are sometimes overlooked in favor of pure imaging.
  • Fusion > Sum of Parts: The biggest "aha moment" is the synergy between modalities. The multimodal approach improved the AUROC by 24.2% over the image-only model and 3.9% over the EMR-only model. This confirms that clinical context significantly aids in the interpretation of medical imagery.

Key Results

PE-MVCNet outperformed several state-of-the-art benchmarks, including the popular PENet and the multimodal PEfusion.

Method AUROC Accuracy F1 Score
3D ResNet101 0.722 0.611 0.701
PENet 0.660 0.623 0.666
PEfusion 0.936 0.882 0.882
PE-MVCNet (Ours) 0.941 0.902 0.906

(Data from Table 1 in the paper)

Ablation studies (Table 2) further proved that the CMAF module specifically contributed to the performance boost, outperforming a "simple fusion" model that lacked the cross-modal attention mechanism.

Ablation studies showing the performance contribution of each module.

Limitations and Future Work

The authors noted that GPU memory capacity constraints forced them to transform features into a lower-dimensional space before fusion, which might result in some information loss.

Future research could explore more sophisticated ways to handle the high dimensionality of 3D medical data or integrate even more diverse data types (like doctor's free-text notes) using Large Language Models (LLMs) to further refine the clinical context.

Summary updated

Abstract

The early detection of a pulmonary embolism (PE) is critical for enhancing patient survival rates. Both image-based and non-image-based features are of utmost importance in medical classification tasks. In a clinical setting, physicians tend to rely on the contextual information provided by Electronic Medical Records (EMR) to interpret medical imaging. However, very few models effectively integrate clinical information with imaging data. To address this shortcoming, we suggest a multimodal fusion methodology, termed PE-MVCNet, which capitalizes on Computed Tomography Pulmonary Angiography imaging and EMR data. This method comprises the Image-only module with an integrated multi-view block, the EMR-only module, and the Cross-modal Attention Fusion (CMAF) module. These modules cooperate to extract comprehensive features that subsequently generate predictions for PE. We conducted experiments using the publicly accessible Stanford University Medical Center dataset, achieving an AUROC of 94.1%, an accuracy rate of 90.2%, and an F1 score of 90.6%. Our proposed model outperforms existing methodologies, corroborating that our multimodal fusion model excels compared to models that use a single data modality. Our source code is available at https://github.com/LeavingStarW/PE-MVCNET.
BibTeX
@inproceedings{guo2024pe,
  author = {Zhaoxin Guo and Zhipeng Wang and Ruiquan Ge and Jianxun Yu and Feiwei Qin and Yuan Tian and Yuqing Peng and Yonghong Li and Changmiao Wang},
  title = {{PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction}},
  booktitle = {ISBI},
  year = {2024},
  doi = {10.1109/isbi56570.2024.10635747},
}
Figure