mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, Yefeng Zheng

Abstract

Accurate brain tumor segmentation typically relies on the availability of multiple MRI modalities, such as T1, T1-contrast (T1c), T2, and FLAIR. However, in clinical practice, it is common for one or more of these modalities to be missing due to scan corruption, patient discomfort, or varying hospital protocols. Existing methods often struggle with such incomplete data, either requiring computationally expensive image synthesis or the training of numerous combinatorial models. This paper presents mmFormer, a unified Transformer-based framework designed to maintain high segmentation accuracy across any combination of available modalities. By integrating hybrid modality-specific encoders with a modality-correlated Transformer, mmFormer achieves state-of-the-art performance, including a 19.07% improvement in Dice score over previous methods when only a single modality is available.

Introduction

In neuro-oncology, multi-contrast MRI is the gold standard for delineating tumor boundaries. Different sequences provide complementary information: T1c is essential for identifying the enhancing tumor core, while FLAIR is superior for detecting peritumoral edema. While deep learning models, particularly Convolutional Neural Networks (CNNs), have excelled at this task, they generally assume a complete set of input modalities. When data is missing, these models often suffer from significant performance degradation.

Previous attempts to address this "incomplete multimodal" problem have utilized generative adversarial networks (GANs) to synthesize missing images or employed knowledge distillation to transfer information from a "full-modality" teacher to a "missing-modality" student. However, these approaches are often computationally intensive or require complex training pipelines. mmFormer addresses these challenges by using a single, unified architecture that leverages the global context-awareness of Transformers to bridge the gap between missing and available data.

Methodology

1. Hybrid Modality-specific Encoders

The mmFormer architecture begins with four independent encoders, one for each MRI modality. To capture both fine-grained local textures and long-range spatial dependencies, each encoder is a "hybrid" structure. It utilizes CNN layers for local feature extraction and an Intra-modal Transformer to model global context within that specific modality. This ensures that even if only one modality is present, the model extracts the most comprehensive features possible from that single source.

2. Modality-correlated Encoder (Inter-modal Transformer)

The core innovation of mmFormer is the Inter-modal Transformer. This component takes the features from all available encoders and models the correlations between them. By applying self-attention across the different modality streams, the model learns a "modality-invariant" representation. This allows the network to infer missing semantic information by looking at related cues in the available modalities—for instance, using T2 features to compensate for a missing FLAIR sequence.

3. Robust Training Strategies

To ensure the model is truly robust to missing data, the authors implemented two key strategies during training:

  • Modality-level Dropout: During the training phase, modalities are randomly "dropped" (replaced with zero vectors) based on a Bernoulli distribution. This forces the network to learn to segment the tumor regardless of which specific subset of data is provided.
  • Auxiliary Regularizers: To prevent the model from becoming overly dependent on cross-modal correlations, a shared-weight decoder acts as a regularizer. It forces each individual encoder to attempt a preliminary segmentation, ensuring that each modality-specific branch remains highly discriminative.

Experiments and Results

Dataset

The model was evaluated using the BraTS 2018 dataset, which contains 285 multi-contrast MRI scans. The evaluation focused on three nested subregions: the Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET).

Performance Benchmarks

mmFormer was compared against established unified models such as HeMIS, U-HVED, and ACN. The results demonstrated significant advantages:

  • Single Modality Robustness: In scenarios where only one MRI modality was available, mmFormer outperformed existing state-of-the-art models by an average of 19.07% in Dice score for the Enhancing Tumor region.
  • Full Modality Excellence: Even when all four modalities were present, mmFormer outperformed TransBTS, a leading Transformer-based model for full-data scenarios. This suggests that the inter-modal Transformer improves feature alignment even when data is complete.
  • Training Efficiency: Compared to knowledge distillation methods like ACN, which can require up to 672 hours of training for teacher and student models, mmFormer is a unified model that reaches convergence in approximately 25 hours.

Discussion and Conclusion

The success of mmFormer highlights the importance of global context in medical imaging. While CNNs are limited by their local receptive fields, the Transformer's ability to relate distant pixels and different modalities allows it to "fill in the blanks" when data is missing.

Despite its performance, the model's complexity (106M parameters) remains a consideration for deployment in resource-constrained clinical environments. Future work may focus on model compression techniques or extending this architecture to other multimodal tasks, such as PET/CT fusion. Ultimately, mmFormer represents a significant step toward making AI-driven diagnostic tools more resilient to the data inconsistencies found in real-world clinical practice.

Summary updated

BibTeX
@inproceedings{zhang2022mmformer,
  author = {Yao Zhang and Nanjun He and Jiawei Yang and Yuexiang Li and Dong Wei and Yawen Huang and Yang Zhang and Zhiqiang He and Yefeng Zheng},
  title = {{mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=kpFL8V5Z2hm},
}
Figure