3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

Rui Nian, Guoyao Zhang, Yao Sui, Yuqi Qian, Qiuying Li, Mingzhang Zhao, Jianhui Li, Ali Gholipour, Simon K. Warfield

Abstract

Magnetic resonance imaging (MRI) is critically important for brain mapping in both scientific research and clinical studies. Precise segmentation of brain tumors facilitates clinical diagnosis, evaluations, and surgical planning. Deep learning has recently emerged to improve brain tumor segmentation and achieved impressive results. Convolutional architectures are widely used to implement those neural networks. By the nature of limited receptive fields, however, those architectures are subject to representing long-range spatial dependencies of the voxel intensities in MRI images. Transformers have been leveraged recently to address the above limitations of convolutional networks. Unfortunately, the majority of current Transformers-based methods in segmentation are performed with 2D MRI slices, instead of 3D volumes. Moreover, it is difficult to incorporate the structures between layers because each head is calculated independently in the Multi-Head Self-Attention mechanism (MHSA). In this work, we proposed a 3D Transformer-based segmentation approach. We developed a Fusion-Head Self-Attention mechanism (FHSA) to combine each attention head through attention logic and weight mapping, for the exploration of the long-range spatial dependencies in 3D MRI images. We implemented a plug-and-play self-attention module, named the Infinite Deformable Fusion Transformer Module (IDFTM), to extract features on any deformable feature maps. We applied our approach to the task of brain tumor segmentation, and assessed it on the public BRATS datasets. The experimental results demonstrated that our proposed approach achieved superior performance, in comparison to several state-of-the-art segmentation methods.

Introduction

Accurate brain tumor segmentation is vital for surgical planning and diagnosis, but it is notoriously difficult because tumors vary wildly in size, shape, and location. Traditional Convolutional Neural Networks (CNNs), like U-Net, are the industry standard but suffer from "local blindness"—their receptive fields are limited, making it hard to capture long-range spatial relationships across the brain.

Standard convolutional block approach (A) for brain tumor segmentation.

While Transformers were introduced to solve this via "self-attention," most existing medical versions process MRI data as 2D slices, which discards crucial 3D volumetric information. Furthermore, standard Multi-Head Self-Attention (MHSA) treats each "attention head" independently, failing to capture the complex structural relationships between different layers of feature extraction.

Methods: The 3D Brainformer Architecture

The proposed 3D Brainformer utilizes a 3D U-Net-like encoder-decoder structure enhanced by two primary innovations:

  1. Fusion-Head Self-Attention (FHSA): Unlike standard Transformers where attention heads work in isolation, FHSA uses "attention logic" and "weight mapping" to combine the outputs of different heads. This allows the model to explore inter-head dependencies and better map long-range spatial relationships in 3D space.
  2. Infinite Deformable Fusion Transformer Module (IDFTM): This is a "plug-and-play" module inserted into the decoder. It is designed to extract features from deformable feature maps, allowing the network to adapt to the irregular and "fuzzy" boundaries of tumors.
Proposed 3D Brainformer approach (B) utilizing the Infinite Deformable Fusion Transformer Module (IDFTM).
  1. Hybrid Decoder: The decoder cascades the IDFTM with a Residual Basic Convolution Module (RBM). This creates a "best of both worlds" scenario: the Transformer components handle the global layout (like edema regions), while the CNN components focus on fine-grained local details (like the tumor core).

Experimental Evaluation

The model was trained and evaluated using the BraTS (Multimodal Brain Tumor Segmentation Challenge) datasets from 2017, 2018, and 2019. These datasets include four MRI modalities: T1, T1ce, T2, and FLAIR. The segmentation targets three sub-regions: the Whole Tumor (WT), the Tumor Core (TC), and the Enhancing Tumor (ET).

Key Results

The 3D Brainformer was compared against state-of-the-art models like 3D U-Net, V-Net, and UNETR.

  • Quantitative Performance: On the BraTS 2018 dataset, the model achieved a Dice score of 0.892 for the Whole Tumor, 0.859 for the Tumor Core, and 0.806 for the Enhancing Tumor. It consistently outperformed UNETR across almost all metrics, including Sensitivity and Hausdorff distance.
  • Qualitative Insights: Visual results showed that the Brainformer is significantly better at depicting the "fuzzy" boundaries of edema (ED) regions. Thermal mapping revealed that the RBM (convolutional) parts of the decoder focused heavily on the Tumor Core, while the IDFTM (transformer) parts focused on the Edema and Enhancing Tumor regions.

Conclusion

The 3D Brainformer successfully bridges the gap between global context and local anatomical details. By breaking head independence in the attention mechanism and introducing deformable modules, it achieves superior performance on complex 3D medical imaging tasks. While 3D volumetric processing remains resource-intensive, the model's efficiency (2–4 minutes per subject) makes it a promising candidate for clinical workflows.

Summary updated

Abstract

Magnetic resonance imaging (MRI) is critically important for brain mapping in both scientific research and clinical studies. Precise segmentation of brain tumors facilitates clinical diagnosis, evaluations, and surgical planning. Deep learning has recently emerged to improve brain tumor segmentation and achieved impressive results. Convolutional architectures are widely used to implement those neural networks. By the nature of limited receptive fields, however, those architectures are subject to representing long-range spatial dependencies of the voxel intensities in MRI images. Transformers have been leveraged recently to address the above limitations of convolutional networks. Unfortunately, the majority of current Transformers-based methods in segmentation are performed with 2D MRI slices, instead of 3D volumes. Moreover, it is difficult to incorporate the structures between layers because each head is calculated independently in the Multi-Head Self-Attention mechanism (MHSA). In this work, we proposed a 3D Transformer-based segmentation approach. We developed a Fusion-Head Self-Attention mechanism (FHSA) to combine each attention head through attention logic and weight mapping, for the exploration of the long-range spatial dependencies in 3D MRI images. We implemented a plug-and-play self-attention module, named the Infinite Deformable Fusion Transformer Module (IDFTM), to extract features on any deformable feature maps. We applied our approach to the task of brain tumor segmentation, and assessed it on the public BRATS datasets. The experimental results demonstrated that our proposed approach achieved superior performance, in comparison to several state-of-the-art segmentation methods.
BibTeX
@misc{nian2023d,
  author = {Rui Nian and Guoyao Zhang and Yao Sui and Yuqi Qian and Qiuying Li and Mingzhang Zhao and Jianhui Li and Ali Gholipour and Simon K. Warfield},
  title = {{3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation}},
  year = {2023},
  eprint = {2304.14508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.14508},
}
Figure