CMIM: Cross-Modal Information Maximization For Medical Imaging

Tristan Sylvain, Francis Dutil, Tess Berthier, Lisa Di-Jorio, Margaux Luck, R. Devon Hjelm, Yoshua Bengio

TL;DR

CMIM is a framework that uses mutual information maximization during training to make medical AI models more robust when data is missing at test time. By forcing different modalities (like MRIs and radiology reports) to share information during the learning phase, the model can perform high-quality classification or segmentation even when only a single, "weak" modality is available for inference.

What Problem Does This Paper Address?

In a perfect clinical world, every patient would have a full suite of diagnostic data: multiple MRI sequences, CT scans, and detailed radiology reports. In reality, medical data is "siloed" and often incomplete. A patient might miss a specific scan due to cost, time constraints, or equipment calibration issues.

This creates a "modality dropping" problem for deep learning. Most multi-modal models struggle when they are trained on several inputs but only receive one at test time. The authors aim to bridge this gap by ensuring that the representations learned from a single modality are as informative as possible, effectively "borrowing" discriminative power from the other modalities present during training.

Methods

The authors introduce Cross-Modal Information Maximization (CMIM). The core idea is to maximize the Mutual Information (MI) between the representations of different modalities during training.

Key technical components include:

  • MI Estimation: Since MI is notoriously difficult to calculate for high-dimensional neural network features, they use the Jensen-Shannon estimator (inspired by MINE and Deep InfoMax) to provide a lower bound on MI.
  • Multi-Level Alignment: They don't just compare global summaries. They define three types of losses:
  • Local-Local ($L^{l \to l}$): Aligning fine-grained spatial features between modalities.
  • Local-Global ($L^{l \to g}$): Aligning local features of one modality with the global summary of another.
  • Global-Global ($L^{g \to g}$): Aligning the overall high-level concepts.
  • Task-Specific Integration: These MI losses are added to standard supervised losses (like cross-entropy for classification or Dice loss for segmentation).
  • Architectures: They utilize standard backbones—U-Net for segmentation and ResNet50 for images—combined with GloVe embeddings for text, proving the framework is flexible across different architectures.

Datasets

The researchers evaluated CMIM on two distinct tasks:

  1. Open-I: A publicly available dataset of chest X-rays paired with radiology reports. The task is multi-label classification (identifying 14 different findings).
  2. BRATS-2015: A brain MRI dataset containing four modalities (FLAIR, T1W, T1C, and T2). The task is voxel-level segmentation of "enhancing" tumors.

Key Insights and Ideas

  • The "Teacher" Effect: CMIM allows "stronger" modalities to act as teachers for "weaker" ones. If one MRI sequence is naturally better at showing a tumor, the MI loss forces the other sequences to extract features that correlate with that "strong" signal.
  • Robustness to Missing Data: By design, the model learns a single-modal representation that is "aware" of what the other modalities would have shown. This makes the model significantly more resilient when those other modalities are dropped at test time.
  • Local vs. Global Trade-offs: The authors found that for spatial tasks like segmentation, local-local information is the most critical. For high-level tasks like classification, mixing local and global information provides the best boost.
  • Overcoming Small Datasets: In the Open-I task, CMIM outperformed TieNet (a state-of-the-art text-image model). The authors argue that CMIM prevents the model from "cheating" by over-relying on text keywords, forcing it to find meaningful visual-textual correlations instead.

Key Results

  • Classification (Open-I): CMIM achieved an AUC of 0.793, outperforming a standard ResNet (0.785) and the specialized TieNet (0.741) when only the image was available at test time.
  • Segmentation (BRATS): The gains were most dramatic on "weaker" modalities. For the FLAIR (F) modality, CMIM achieved a Dice score of 23.37%, compared to just 5.57% for the state-of-the-art HeMIS baseline.
  • Consistency: Across almost all test-time scenarios where only one modality was present, CMIM either set a new state-of-the-art or was a very close second.

Limitations and Future Work

  • Extreme Dropping: This paper specifically focused on the "extreme" case where only one modality is available at test time. The authors note that future work should explore how the model behaves when varying combinations of modalities (e.g., 2 out of 4) are available.
  • Zero-Shot Potential: The authors suggest that this setup could pave the way for zero-shot learning, where a model could potentially handle entirely new modalities at test time if they share enough mutual information with the training set.
  • Hyperparameter Sensitivity: The $\lambda$ weights for the different MI losses require tuning, which might vary significantly between different medical imaging tasks.

Summary updated

Abstract

In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time.In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.
BibTeX
@inproceedings{sylvain2021cmim,
  author = {Tristan Sylvain and Francis Dutil and Tess Berthier and Lisa Di-Jorio and Margaux Luck and R. Devon Hjelm and Yoshua Bengio},
  title = {{CMIM: Cross-Modal Information Maximization For Medical Imaging}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414132},
}

Loading related content...

Figure