MISSU: 3D Medical Image Segmentation via Self-Distilling TransUNet

Nan Wang, Shaohui Lin, Xiaoxiao Li, Ke Li, Yunhang Shen, Yue Gao, Lizhuang Ma

TL;DR

MISSU is a hybrid 3D medical image segmentation framework that combines the local feature extraction of CNNs with the global modeling power of Transformers. Its core innovation is a self-distillation mechanism that "teaches" the encoder to capture rich, multi-scale local details during training, which are then retained implicitly at inference time without any extra computational cost.

What Problem Does This Paper Address?

In 3D medical imaging (like MRI or CT scans), precise segmentation is a balancing act.

  1. The Receptive Field Gap: Traditional U-Nets (CNNs) are excellent at capturing local textures but have a limited "receptive field," making it hard for them to understand global context (e.g., how a tumor relates to distant organs).
  2. The Transformer Trade-off: Transformers excel at modeling long-range dependencies but are computationally expensive for high-resolution 3D data. Furthermore, they often lose the fine-grained spatial details (edges and boundaries) that CNNs handle so well.
  3. The Efficiency Bottleneck: Multi-path networks that try to solve both problems simultaneously are often too slow or memory-intensive for practical clinical use.

The authors tackle the question: How can we design a unified framework that models both global context and local detail during training while remaining efficient during inference?

Methods

The proposed architecture, MISSU, follows an encoder-decoder structure with three key components:

  • The Hybrid Encoder: The model starts with a 3D CNN to extract local features and down-sample the image. These features are then "tokenized" and fed into a Transformer block to model global, long-range correlations.
  • Multi-Scale Fusion (MSF) Block: This is a "teacher" module composed of parallel atrous (dilated) convolutions with different rates. It captures features at multiple scales to refine fine-grained details like organ borders.
  • Self-Distillation (The "Secret Sauce"): During training, the MSF block generates high-quality, multi-scale features. A distillation loss ($L_{sd}$) forces the main CNN encoder to mimic these rich features.
  • Computation-Free Inference: Because the knowledge from the MSF block is "distilled" into the main encoder's weights, the MSF block is discarded during testing. This allows the model to achieve high accuracy with the speed of a much simpler network.

Datasets

The authors evaluated MISSU on two major benchmarks:

  • BraTS 2019: A brain tumor segmentation dataset containing 335 cases with four MRI modalities (T1, T1ce, T2, FLAIR). It focuses on three regions: Whole Tumor (WT), Tumor Core (TC), and Enhanced Tumor (ET).
  • CHAOS: A dataset for liver segmentation using T1 and T2 MRI sequences.

Key Insights and Ideas

  • Implicit vs. Explicit Fusion: Most hybrid models try to explicitly fuse local and global paths at every step. MISSU proves that implicit fusion via distillation is highly effective, allowing the model to "remember" local details without needing the extra hardware-heavy paths at runtime.
  • The Power of Atrous Convolutions: By using atrous convolutions in the MSF block, the model can expand its field of view to capture multi-scale information without losing spatial resolution or increasing parameter counts excessively.
  • Efficiency as a First-Class Citizen: The paper emphasizes the "trade-off between positioning accuracy and multi-path information." By moving the complexity to the training phase, they achieve a state-of-the-art (SOTA) performance-to-complexity ratio.

Key Results

MISSU significantly outperformed both pure CNNs (like nnUNet) and other Transformer-based models (like TransBTS and UNETR):

  • Accuracy (BraTS 2019): Achieved 89.98% (WT), 85.77% (TC), and 80.14% (ET) Dice scores, setting a new SOTA.
  • Accuracy (CHAOS): Achieved a 96.78% Dice score for liver segmentation.
  • Efficiency: Despite its high accuracy, MISSU is remarkably lean. It uses only 10.50M parameters and 132.11 GFLOPs. For comparison, the standard 3D UNet requires 1,670 GFLOPs, and the competing TransUNet has nearly 10 times the parameters (96.07M).
  • Qualitative Edge: Visualizations (Grad-CAM) showed that the Transformer block successfully focused on global structure while the MSF-distilled encoder accurately captured the irregular boundaries of lesions.

Limitations and Future Work

  • Data Modalities: While the model performs exceptionally on MRI, its performance on other modalities like CT or ultrasound wasn't the primary focus of this study.
  • Metadata Integration: The authors suggest that future iterations could integrate non-image data (such as patient age or sex) directly into the Transformer tokens to further improve diagnostic accuracy.
  • Hyperparameter Sensitivity: As shown in the ablation studies, the choice of embedding dimension ($d$) and the number of Transformer layers ($L$) is critical; more is not always better, as it can lead to irrelevant embeddings.

Summary updated

Abstract

U-Nets have achieved tremendous success in medical image segmentation. Nevertheless, it may have limitations in global (long-range) contextual interactions and edge-detail preservation. In contrast, the Transformer module has an excellent ability to capture long-range dependencies by leveraging the self-attention mechanism into the encoder. Although the Transformer module was born to model the long-range dependency on the extracted feature maps, it still suffers high computational and spatial complexities in processing high-resolution 3D feature maps. This motivates us to design an efficient Transformer-based UNet model and study the feasibility of Transformer-based network architectures for medical image segmentation tasks. To this end, we propose to self-distill a Transformer-based UNet for medical image segmentation, which simultaneously learns global semantic information and local spatial-detailed features. Meanwhile, a local multi-scale fusion block is first proposed to refine fine-grained details from the skipped connections in the encoder by the main CNN stem through self-distillation, only computed during training and removed at inference with minimal overhead. Extensive experiments on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best performance over previous state-of-the-art methods. Code and models are available at: https://github.com/wangn123/MISSU.git
BibTeX
@article{wang2023missu,
  author = {Nan Wang and Shaohui Lin and Xiaoxiao Li and Ke Li and Yunhang Shen and Yue Gao and Lizhuang Ma},
  title = {{MISSU: 3D Medical Image Segmentation via Self-Distilling TransUNet}},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2023},
  url = {https://openreview.net/forum?id=Ngc7RK0tmE},
}
Figure