TL;DR
This paper introduces a novel Transformer-based architecture (TriMF) for fusing medical data from multiple sources (X-rays, radiology reports, and tabular data) to improve disease diagnosis. Crucially, TriMF is designed to be robust to missing data modalities, a common challenge in real-world clinical settings, by employing a multi-modal fusion framework and a specialized contrastive loss function.
What Problem Does This Paper Address?
Deep learning models have shown great promise in medical data analysis, but their performance often relies on having complete data from all relevant sources (modalities). In clinical practice, it's very common for certain data modalities to be missing for a given patient due to various reasons. This "missing modality" problem significantly degrades the performance of traditional multi-modal fusion models. Existing solutions like data imputation can be complex and require massive datasets. This paper tackles this challenge by proposing a model that can inherently handle missing modalities without needing to generate or impute the missing data.
Methods
The core of the proposed solution is a Transformer-based tri-modal fusion (TriMF) architecture. Here's a breakdown of the key components:
- Feature Embedding: Each modality (X-ray images, radiology reports, tabular patient data) is first processed by modality-specific embedding networks to extract meaningful features.
- Images: A pre-trained DenseNet-121 is used.
- Text: PubMedBERT, a model pre-trained on biomedical text, is employed.
- Tabular Data: A shallow neural network is used.
- Bi-modal Fusion Module (BiMF): The TriMF architecture is built upon several BiMF modules. Each BiMF module fuses two modalities using a combination of:
- Self-Attention (SA): Learns relationships within a single modality.
- Co-Attention (CA): Learns pairwise relationships between two modalities.
- Low-Rank Multi-modal Fusion (LMF): A technique to efficiently fuse the attention outputs. These BiMF modules are stacked to create a "deep BiMF encoder."
- Tri-modal Fusion Framework: Three BiMF modules are used to create pairwise fusions (image+text, image+tabular, text+tabular). The outputs of these BiMF modules are then combined (simply summed in this case) to form the final tri-modal representation.
- Multivariate Loss Function (FRCL): To enhance robustness to missing modalities, a novel Fusion Representation Contrastive Loss (FRCL) is introduced. This loss function encourages the fused representation of a complete set of modalities to be similar to the fused representation of a subset of modalities. This helps the model learn representations that are less sensitive to the presence or absence of specific data sources.
Datasets
The study utilizes two publicly available datasets:
- MIMIC-IV: Used for patient demographics and laboratory test data (tabular modality).
- MIMIC-CXR: Used for chest radiographs (image modality) and associated radiology reports (text modality).
The final experimental dataset comprised 23,421 samples with complete information across all three modalities.
Key Insights and Ideas
- Robustness to Missing Data is Crucial: The paper strongly emphasizes that real-world medical data is rarely complete, and models must be designed to handle this.
- Transformer Architecture for Fusion: Leveraging the power of Transformers, which have excelled in NLP and vision, for multi-modal medical data fusion is a promising direction.
- Bi-modal Fusion as Building Blocks: Decomposing the tri-modal fusion problem into a series of bi-modal fusions simplifies the architecture and allows for graceful degradation when modalities are missing.
- Contrastive Loss for Missing Modalities: The FRCL function is a key innovation, directly addressing the performance drop caused by missing data by making representations consistent across different modality combinations.
- Text Modality Dominance: The text modality (radiology reports) appears to be the most informative for the disease classification task in this study.
Key Results
- Superior Performance: The proposed TriMF model outperformed the state-of-the-art MedViLL model on 11 out of 14 disease labels, achieving an average AUROC of 0.914 and AUPRC of 0.552.
- Robustness Demonstrated: When tested on incomplete modality sets, the TriMF model showed only a slight degradation in performance. The average AUROC decreased by only 0.002, 0.071, and 0.003 when tabular, text, and image modalities were missing, respectively.
- Impact of Missing Modalities: The absence of the text modality led to the most significant performance drop, highlighting its importance. Conversely, missing tabular data had the least impact.
- Ablation Study Success: Removing key components like SA, LMF, or the FRCL function led to a decrease in overall performance, validating their importance in the TriMF architecture.
Limitations and Future Work
- Limited Modalities: The study focused on three specific modalities. Future work could explore fusing more modalities.
- Scalability Concerns: While the architecture can fuse more modalities by adding BiMF modules, this could lead to a significant increase in parameters.
- Submodality Variety: The study used a limited set of submodalities (e.g., X-ray, not CT/MRI; radiology reports, not pathology reports). Exploring a wider range of submodalities could introduce more complex missing modality problems.
- Dataset Specificity: Experiments were conducted on an English public dataset. Validation on datasets from different linguistic and cultural backgrounds (e.g., Chinese datasets) is needed.
- Real-world Clinical Application: The proposed architecture should be validated in real clinical scenarios.
Summary updated