Abstract
In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.
Introduction
In clinical practice, diagnostic data is often incomplete. While a comprehensive patient profile might include multiple MRI sequences, CT scans, and radiology reports, real-world constraints like cost, time, and equipment availability often lead to "modality dropping." Deep learning models trained on multi-modal data typically struggle when one or more inputs are missing at inference time.
The Cross-Modal Information Maximization (CMIM) framework addresses this by ensuring that representations learned from a single modality are as informative as possible, effectively "borrowing" discriminative power from other modalities present during training.
Methodology
CMIM leverages Mutual Information (MI) maximization between the representations of different modalities during the training phase. Since MI is difficult to calculate directly for high-dimensional neural network features, the framework employs the Jensen-Shannon estimator to provide a lower bound on MI.
The framework defines three levels of alignment:
- Local-Local ($L^{l \to l}$): Aligns fine-grained spatial features between modalities.
- Local-Global ($L^{l \to g}$): Aligns local features of one modality with the global summary of another.
- Global-Global ($L^{g \to g}$): Aligns high-level conceptual summaries.
These MI losses are integrated with standard supervised losses, such as cross-entropy for classification or Dice loss for segmentation. The architecture is flexible, utilizing standard backbones like U-Net for segmentation, ResNet50 for images, and GloVe embeddings for text.
Experimental Setup
The researchers evaluated CMIM on two distinct tasks:
- Multi-label Classification (Open-I): Chest X-rays paired with radiology reports to identify 14 findings.
- Voxel-level Segmentation (BRATS-2015): Brain MRI dataset with four modalities (FLAIR, T1W, T1C, and T2) to segment "enhancing" tumors.
Results and Discussion
CMIM demonstrates a "Teacher Effect," where stronger modalities guide the learning of weaker ones. By forcing mutual information maximization, the model learns single-modal representations that are "aware" of the missing data's characteristics.
Classification Performance
On the Open-I dataset, CMIM achieved an AUC of 0.793 when only images were available at test time, outperforming both a standard ResNet (0.785) and the specialized TieNet (0.741). This suggests that CMIM prevents over-reliance on text keywords, forcing the model to find meaningful visual-textual correlations.
Segmentation and Modality Dropping
The most significant gains were observed in the segmentation of "weaker" MRI modalities, where CMIM significantly outperformed state-of-the-art baselines.
As shown in the results, for the FLAIR (F) modality, CMIM achieved a Dice score of 23.37%, a substantial improvement over the 5.57% achieved by the HeMIS baseline. Across nearly all scenarios where only one modality was present at inference, CMIM either established a new state-of-the-art or performed competitively.
Conclusion
CMIM provides a robust solution for medical imaging tasks where data completeness cannot be guaranteed at test time. While the current study focuses on the extreme case of single-modality inference, future work will explore performance across various combinations of missing modalities and the potential for zero-shot learning.
Summary updated