MM-UNet: A multimodality brain tumor segmentation network in MRI images

Liang Zhao, Jiajun Ma, Yu Shao, Chaoran Jia, Jingyuan Zhao, Yuan HongFrontiers in Oncology 2022 · DOI

TL;DR

MM-UNet is a novel deep learning architecture designed to improve brain tumor segmentation by fusing data from multiple MRI modalities (T1, T1c, T2, and FLAIR). By combining a multi-encoder structure with a hybrid attention mechanism and dilated convolutions, the model achieves a mean Dice score of 79.2% on the BraTS 2020 dataset, significantly outperforming standard U-Net baselines.

What Problem Does This Paper Address?

Accurate brain tumor segmentation is critical for diagnosis and treatment planning, yet manual segmentation by radiologists is slow, labor-intensive, and prone to subjective bias. While automated deep learning models like U-Net have become the standard, they face two major hurdles in clinical settings:

  1. Modality Limitations: A single MRI sequence (like T1) often lacks enough information to distinguish between different tumor tissues (e.g., edema vs. necrotic core).
  2. Information Loss: Standard "downsampling" (pooling) layers in neural networks help the model understand the "big picture" but often "blur" the fine details of tumor edges, which are vital for surgical precision.

The authors introduce MM-UNet to solve these issues by effectively merging features from different MRI sequences and preserving spatial resolution through smarter convolution techniques.

Methods

The MM-UNet architecture evolves the classic U-Net "U-shape" with several key innovations:

  • Multi-Encoder, Single-Decoder: Instead of feeding all MRI modalities into one input, the model uses four independent encoders—one for each modality. This allows the network to extract unique low-level features from T1, T1c, T2, and FLAIR separately before fusing them.
  • Hybrid Attention Block (HAB): To filter out noise and focus on relevant features, the authors developed the HAB. It applies attention across both channels (which modality/feature is most important?) and space (where in the brain is the tumor?). Notably, they use "range pooling" (max minus min) alongside standard average and max pooling to capture better contrast information.
  • Dilated Convolution Block (DCB): Placed at the bottom of the "U," the DCB uses "atrous" (dilated) convolutions. This allows the model to expand its receptive field—seeing a larger area of the brain—without losing the fine-grained pixel resolution that usually disappears during standard pooling.
  • Hybrid Loss Function: The model uses a weighted combination of Dice loss (for overlap accuracy) and Focal loss (to handle the "class imbalance" problem where the tumor is much smaller than the surrounding healthy brain tissue).

Datasets

The researchers used the BraTS 2020 (Brain Tumor Segmentation Challenge) dataset:

  • Training Set: 369 patients.
  • Validation Set: 125 patients.
  • Data Types: Each patient record includes four MRI modalities (T1, T1c, T2, and FLAIR).
  • Labels: The model segments three nested regions: the Enhanced Tumor (ET), the Tumor Core (TC), and the Whole Tumor (WT).

Key Insights and Ideas

  • The Power of Range Pooling: While most models rely on average or max pooling, the authors found that adding "range pooling" helps the attention mechanism better identify areas of high contrast, which is a hallmark of tumor boundaries in MRI.
  • Serial Attention Wins: Through experimentation, the authors discovered that applying channel attention followed by spatial attention (serially) produced better results than applying them in parallel. This suggests that the model benefits from refining "what" to look at before deciding "where" to look.
  • Dilated Convolutions as a "Bridge": By using the DCB as a bridge between the encoder and decoder, the model successfully captures multi-scale information, allowing it to detect both very large tumor masses and smaller, isolated lesions with high fidelity.

Key Results

MM-UNet was benchmarked against U-Net, Attention U-Net, and ResUNet. The quantitative improvements were consistent across the board:

  • Mean Dice Score: 79.2% (an improvement over the 70-73% range seen in basic U-Net variants for specific sub-regions).
  • Hausdorff Distance (95%): Achieved a mean of 8.466. This metric measures how far the predicted boundary is from the ground truth; a lower number indicates much sharper, more accurate edges.
  • Ablation Success: Removing either the HAB or the DCB led to a drop in performance, proving that both the attention mechanism and the dilated convolutions are essential to the model's success.

Limitations and Future Work

Despite the strong results, the authors noted a few areas for improvement:

  • Sensitivity to Positioning: The effectiveness of the Dilated Convolution Block (DCB) depends heavily on where it is placed in the network hierarchy.
  • Small Area Neglect: While the model is excellent at large-scale perception, it still occasionally ignores very tiny tumor details in favor of the primary mass.
  • Computational Complexity: Using four separate encoders increases the parameter count (though the authors managed to keep this increase relatively modest at ~111M parameters).

Future research may focus on optimizing the placement of these blocks or exploring even more efficient ways to fuse modalities without increasing the computational load.

Summary updated

Abstract

The global annual incidence of brain tumors is approximately seven out of 100,000, accounting for 2% of all tumors. The mortality rate ranks first among children under 12 and 10th among adults. Therefore, the localization and segmentation of brain tumor images constitute an active field of medical research. The traditional manual segmentation method is time-consuming, laborious, and subjective. In addition, the information provided by a single-image modality is often limited and cannot meet the needs of clinical application. Therefore, in this study, we developed a multimodality feature fusion network, MM-UNet, for brain tumor segmentation by adopting a multi-encoder and single-decoder structure. In the proposed network, each encoder independently extracts low-level features from the corresponding imaging modality, and the hybrid attention block strengthens the features. After fusion with the high-level semantic of the decoder path through skip connection, the decoder restores the pixel-level segmentation results. We evaluated the performance of the proposed model on the BraTS 2020 dataset. MM-UNet achieved the mean Dice score of 79.2% and mean Hausdorff distance of 8.466, which is a consistent performance improvement over the U-Net, Attention U-Net, and ResUNet baseline models and demonstrates the effectiveness of the proposed model.
BibTeX
@inproceedings{zhao2022mm,
  author = {Liang Zhao and Jiajun Ma and Yu Shao and Chaoran Jia and Jingyuan Zhao and Yuan Hong},
  title = {{MM-UNet: A multimodality brain tumor segmentation network in MRI images}},
  booktitle = {Frontiers in Oncology},
  year = {2022},
  doi = {10.3389/fonc.2022.950706},
}
Figure