MedMAP: Promoting Incomplete Multi-Modal Brain Tumor Segmentation With Alignment

Tianyi Liu, Zhaorui Tan, Muyin Chen, Xi Yang, Haochuan Jiang, Kaizhu Huang

TL;DR

MedMAP is a novel framework designed to solve the "missing modality" problem in brain tumor segmentation, where one or more MRI scans (like T1 or Flair) are unavailable. By mathematically aligning the latent features of different imaging modalities to a shared "distribution anchor," MedMAP helps AI models learn universal tumor characteristics, leading to significant accuracy gains in real-world clinical scenarios.

What Problem Does This Paper Address?

Accurate brain tumor segmentation typically requires four distinct MRI modalities: T1, T1-weighted contrast-enhanced (T1ce), T2, and Fluid Attenuation Inversion Recovery (Flair). Each provides unique information—for example, T2 is great for seeing edema, while T1ce highlights the tumor core.

However, in clinical practice, it is common for one or more of these scans to be missing due to patient discomfort, limited scanning time, or data corruption. Existing AI approaches like Knowledge Distillation (KD) or Shared Latent Spaces (SLS) attempt to handle missing data, but they often fail to bridge the "modality gap." They treat different scans as separate channels rather than learning the underlying, invariant features of the tumor that exist across all scan types. This leads to a "scattered" latent space where the model struggles to generalize when a specific scan is missing.

Methods

The authors introduce the Medical Modality Alignment Paradigm (MedMAP). Its core innovation is forcing the model to map features from every modality into the same region of mathematical space.

  • Distribution Anchors ($P_{mix}$): Since medical imaging lacks the massive pre-trained models available in natural computer vision (which usually serve as "anchors"), the authors created their own. They propose an Adaptive Alignment strategy where the anchor is a learned, weighted mixture of the distributions of all available modalities.
  • Individual Alignment: A key theoretical contribution of the paper is a proof using information theory. They demonstrate that aligning each modality individually to the anchor provides a "tighter evidence lower bound" (ELBO) than trying to align all modalities collectively. This makes the training process more efficient and the resulting features more robust.
  • Plug-and-Play Compatibility: MedMAP isn't a standalone model but a paradigm. It can be integrated into existing architectures (like U-Nets or Transformers) by simply adding an alignment loss term ($\mathcal{L}_A$) to the standard segmentation loss.

Datasets

The researchers evaluated MedMAP using two gold-standard benchmarks in the field:

  • BraTS 2018: 285 patient cases.
  • BraTS 2020: 369 patient cases. Both datasets include the four standard MRI modalities and ground-truth labels for three tumor regions: Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET).

Key Insights and Ideas

  • The "Aha" Moment on Alignment: The researchers realized that the "modality gap" is essentially a domain adaptation problem. By treating different MRI sequences as different "domains" and forcing them to overlap in the model's "brain" (the latent space), the model stops relying on the specific style of a T2 scan and starts looking for the substance of the tumor.
  • Visualizing the Gap: Using t-SNE plots (which project high-dimensional data into 2D), the authors showed that without MedMAP, features from different modalities form distinct, isolated clusters. With MedMAP, these clusters merge into a single, cohesive "cloud," proving the model has learned modality-invariant features.
  • Theoretical Rigor: Unlike many empirical deep learning papers, this work provides a mathematical proof showing why their specific alignment strategy is superior to group-based alignment.

Key Results

MedMAP was tested across three different state-of-the-art "backbone" models (PMKL, mmFormer, and ACN). The results showed consistent improvements:

  • Accuracy Boost: On the BraTS2018 dataset, MedMAP improved the Dice Score (a measure of overlap between predicted and actual tumor) by 7.67% for the mmFormer model and 3.68% for PMKL.
  • Solving the Hardest Cases: The most significant gains occurred in the most difficult segmentation targets. For the "Enhancing Tumor" (ET) category, which is often the hardest to spot, MedMAP improved performance by over 11% in some scenarios.
  • Extreme Data Scarcity: Even when 3 out of 4 modalities were missing, models equipped with MedMAP performed significantly better than their baseline counterparts, making them much more viable for clinics with limited resources.

Limitations and Future Work

  • Hyperparameter Tuning: The framework relies on a weighting parameter ($\alpha$) for the alignment loss. While the authors found it relatively stable, it still requires empirical setting (they used 0.125).
  • Initialization Bias: The "Adaptive Alignment" requires an initial guess for weights. The authors found that starting with a bias toward the "Flair" modality worked best, suggesting the method still benefits from some human-in-the-loop prior knowledge.
  • Broader Application: The study focused exclusively on brain tumors. Future work could explore whether this alignment paradigm works for other multi-modal tasks, such as combining CT and PET scans for lung cancer or different ultrasound modes for cardiac health.

Summary updated

Abstract

Brain tumor segmentation is often based on multiple magnetic resonance imaging (MRI). However, in clinical practice, certain modalities of MRI may be missing, which presents a more difficult scenario. To cope with this challenge, Knowledge Distillation, Domain Adaption, and Shared Latent Space have emerged as commonly promising strategies. However, recent efforts to address the missing modality problem in brain tumor segmentation typically overlook the modality gaps and thus fail to learn important invariant feature representations across different modalities. Such drawback consequently leads to limited performance for missing modality models. To ameliorate these problems, pre-trained models are used in natural visual segmentation tasks to minimize the gaps. However, promising pre-trained models are difficult to obtain in the brain tumor segmentation task due to the lack of sufficient data. Along this line, in this paper, we propose a novel paradigm that aligns latent features of involved modalities to a well-defined distribution anchor as the substitution of the pre-trained model. As a major contribution, we prove that our novel training paradigm ensures a tight evidence lower bound, thus theoretically certifying its effectiveness. Extensive experiments on different backbones validate that the proposed paradigm can enable invariant feature representations and produce models with narrowed modality gaps. Models with our alignment paradigm show their superior performance on both BraTS2018, BraTS2020 and Brain Metastasis datasets.
BibTeX
@article{liu2026medmap,
  author = {Tianyi Liu and Zhaorui Tan and Muyin Chen and Xi Yang and Haochuan Jiang and Kaizhu Huang},
  title = {{MedMAP: Promoting Incomplete Multi-Modal Brain Tumor Segmentation With Alignment}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2026},
  doi = {10.1109/jbhi.2025.3600496},
}
Figure