Modality-Aware and Shift Mixer for Multi-Modal Brain Tumor Segmentation

Zhongzhen Huang, Linda Wei, Shaoting Zhang, Xiaofan Zhang

Abstract

Accurate brain tumor segmentation is vital for clinical diagnosis and treatment planning. While multi-modal MRI provides complementary information across sequences like T1, T1-CE, T2, and FLAIR, standard deep learning approaches often fail to exploit the specific relationships between these modalities efficiently. We introduce the Modality-Aware and Shift Mixer (MASM), a novel framework designed to explicitly model inter-modality dependencies. By utilizing a Modality-Aware module for low-level feature pruning and a Modality-Shift module for high-level feature mixing, MASM achieves state-of-the-art performance on the BraTS 2021 dataset. Notably, it maintains significantly lower computational costs than existing transformer-based methods, offering a more practical solution for clinical integration.

Introduction

In clinical practice, radiologists do not rely on a single brain scan; they synthesize information from multiple MRI modalities to identify distinct tumor sub-regions. For instance, T2 and FLAIR sequences are particularly effective at highlighting edema, while T1-weighted contrast-enhanced (T1-CE) scans are essential for identifying the tumor core.

Despite this, most current deep learning models employ an "early-fusion" strategy, where different modalities are simply stacked as input channels. This approach has two primary limitations: it overlooks the unique, non-linear correlations between specific modality pairs and fails to effectively exchange information across different spatial scales at deeper network layers. The MASM framework addresses these gaps by mimicking the diagnostic logic used by human experts to "understand" how these modalities relate to one another.

Comparison of standard early-fusion (b) versus the proposed modality-specific segmentation logic (c).

The MASM Framework

MASM is built upon a 3D U-Net backbone with a shared encoder to maintain a manageable parameter count. The core innovation lies in two plug-and-play modules that optimize how information is processed and fused across the network.

Modality-Aware Module (Low-Level)

Operating at the early stages of the network, this module groups modalities into pairs based on neuroimaging logic: T1 with T1-CE (anatomy and blood supply) and T2 with FLAIR (edema and water signals). It employs a masking mechanism to identify and prune redundant patches. Given that a significant portion of a medical scan often contains healthy tissue, this pruning allows the model to focus its computational resources on the most discriminative, tumor-relevant features.

Modality-Shift Module (High-Level)

At the deepest layers where features are highly abstract, MASM introduces a "Modality-Shift" operation. This module uses a mosaic pattern to shift patches across different modalities. This clever architectural design allows standard self-attention mechanisms to process information from multiple modalities simultaneously without adding any extra parameters or computational overhead. This "zero-parameter interaction" enables complex multi-modal reasoning for free.

The detailed architecture of the MASM network, highlighting the integration of Modality-Aware and Modality-Shift modules within the U-Net structure.

Experimental Results

Dataset and Evaluation

The model was rigorously evaluated on the BraTS 2021 (Brain Tumor Segmentation) dataset, which consists of 1,251 MRI scans. Each case includes four modalities (T1, T1-CE, T2, FLAIR) and expert-annotated labels for three nested sub-regions: the Enhancing Tumor (ET), Tumor Core (TC), and Whole Tumor (WT).

Performance and Efficiency

MASM demonstrated superior performance compared to several prominent models, including UNETR, SwinUNETR, and nnU-Net. On the BraTS 2021 test set, MASM achieved an average Dice score of 0.912 and a 95% Hausdorff Distance (HD95) of 5.65.

Beyond accuracy, MASM is exceptionally efficient. It requires only 160.1 GFLOPs, which is approximately 1/4th of the computation required by SwinUNETR (~775 GFLOPs) and 1/7th of UNETR (~1159 GFLOPs). Ablation studies confirmed that both the low-level pruning and high-level shifting modules are critical, as removing either led to a measurable decline in segmentation accuracy.

Discussion and Conclusion

By leveraging domain-informed pairing and efficient architectural interactions, MASM bridges the gap between clinical neuroimaging logic and deep learning. The success of the Modality-Aware module suggests that in multi-modal tasks, pruning redundant data can actually improve the signal-to-noise ratio.

While the current iteration uses fixed shift patterns, future work may explore dynamic patterns that adapt to individual patient scans. Furthermore, while this study focused on brain tumors, the MASM architecture holds promise for other multi-modal medical imaging tasks, such as pelvic or abdominal segmentation. MASM represents a significant step toward high-performance, computationally lean AI tools for the clinical environment.

Summary updated

BibTeX
@inproceedings{huang2024modality,
  author = {Zhongzhen Huang and Linda Wei and Shaoting Zhang and Xiaofan Zhang},
  title = {{Modality-Aware and Shift Mixer for Multi-Modal Brain Tumor Segmentation}},
  booktitle = {ECAI},
  year = {2024},
  doi = {10.3233/faia240502},
}
Figure