Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment

Tianyi Liu, Zhaorui Tan, Haochuan Jiang, Xi Yang, Kaizhu Huang

Introduction

In clinical neuro-oncology, multi-modal Magnetic Resonance Imaging (MRI) is the gold standard for accurate brain tumor segmentation. Typically, four distinct sequences—FLAIR, T1, T1ce, and T2—are utilized to capture different physical properties of brain tissue, such as fluid presence, blood flow, and structural lesions. However, in practical clinical environments, it is common for one or more of these modalities to be missing due to corrupted scans, patient motion, or limited imaging protocols. This "missing modality" problem significantly degrades the performance of standard segmentation models.

The Challenge: The Modality Gap

The prevailing approach to addressing missing data is Knowledge Distillation (KD), where a "teacher" model (trained on all four modalities) guides a "student" model (trained on a subset). However, traditional KD methods often treat different MRI sequences merely as separate input channels, ignoring the inherent statistical differences between them—a phenomenon known as the "modality gap."

Because the teacher model fails to learn a unified, modality-invariant representation, the guidance it provides to the student is often sub-optimal. Without explicit alignment, the latent features of a T1 scan and a T2 scan remain distinct, making it difficult for the student to generalize when certain inputs are absent.

Methodology: Latent Space Alignment

This paper introduces a novel alignment paradigm designed to force the latent features of different modalities into a shared distribution. By ensuring that the "meaning" of a feature is consistent regardless of its source modality, the model becomes significantly more robust to missing data.

The Alignment Paradigm

Instead of allowing the encoder to produce scattered features for each modality, the authors propose forcing these features to align with a shared distribution anchor, denoted as $P_{mix}$. This ensures that the latent space is modality-invariant. The authors explored three distinct types of anchors:

  1. Standard Normal Distribution ($P^N_{mix}$): Similar to a Variational Autoencoder (VAE), this forces features into a Gaussian distribution. Interestingly, this performed the worst, suggesting that the latent space must remain anchored to actual data distributions to preserve segmentation details.
  2. Single Modality Anchor ($P^k_{mix}$): Aligning all features to the distribution of the single most informative modality (identified as T1ce).
  3. Adaptive Alignment ($P^*_{mix}$): A weighted mixture of all modalities where the weights are learned dynamically during training. This approach proved most effective.

Theoretical Foundation

The authors provide a rigorous mathematical foundation for this approach, proving that aligning modalities individually to a shared anchor provides a tighter Evidence Lower Bound (ELBO) than mapping all modalities as a single block. This "single-letterization" approach ensures that the model captures the underlying structure of the tumor more effectively than traditional multi-channel inputs.

Experimental Evaluation

The study utilized the 2018 Brain Tumor Segmentation Challenge (BraTS) dataset, consisting of 285 subjects. The models were evaluated on their ability to segment the Whole Tumor (WT), Tumor Core (TC), and Enhancing Core (EC).

Key Insights

  • The Dominance of T1ce: Through Adaptive Alignment, the authors discovered that the T1ce modality received the highest learned weight (2.2055), while FLAIR received the lowest (0.3055). This confirms that T1ce contains the most critical information for defining the tumor's latent structure.
  • Visualizing Alignment: Using t-SNE visualizations, the authors demonstrated that without alignment, features from different modalities form distinct, separated clusters. With the $P^*_{mix}$ alignment, these clusters overlap almost perfectly, indicating that the model has successfully learned modality-invariant features.

Results and Impact

The proposed paradigm was integrated into several state-of-the-art backbones, including KD-Net, PMKL, ProtoKD, and SMU-Net. The results were consistent across all architectures:

  • Performance Boost: The alignment paradigm achieved an average improvement of 1.75 on the Dice score for student models operating with missing modalities.
  • Teacher Enhancement: Even the teacher model, which has access to all data, saw a 1.03 Dice score improvement simply by incorporating the alignment loss during training.
  • Robustness in Extreme Scenarios: In cases where three out of four modalities were missing, the aligned models significantly outperformed baselines, showing higher sensitivity to small tumor regions and fewer false positives.

Conclusion

By addressing the "modality gap" through latent space alignment, this research provides a more effective framework for missing modality brain tumor segmentation. The transition from treating MRI sequences as simple channels to treating them as distinct distributions that must be aligned represents a significant step forward in making AI-driven diagnostics more resilient to real-world data inconsistencies. Future work may extend this logic to other multi-modal medical tasks, such as cardiac or pelvic imaging.

Summary updated

BibTeX
@inproceedings{liu2024mind,
  author = {Tianyi Liu and Zhaorui Tan and Haochuan Jiang and Xi Yang and Kaizhu Huang},
  title = {{Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822387},
}
Figure