MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

Nan Wang, Shaohui Lin, Xiaoxiao Li, Ke Li, Yunhang Shen, Yue Gao, Lizhuang Ma

Abstract

MISSU is a novel hybrid 3D medical image segmentation framework that synergizes the local feature extraction capabilities of Convolutional Neural Networks (CNNs) with the global modeling strengths of Transformers. Central to its design is a self-distillation mechanism that enables the encoder to learn multi-scale local details during training, which are then retained implicitly during inference. This approach allows MISSU to achieve state-of-the-art performance on benchmarks like BraTS 2019 and CHAOS while maintaining high computational efficiency.

Introduction

Precise segmentation in 3D medical imaging, such as MRI or CT scans, requires balancing local spatial details with global anatomical context. Traditional 3D U-Nets excel at local texture capture but suffer from a limited receptive field. Conversely, Transformers offer superior long-range dependency modeling but are computationally intensive for high-resolution 3D data and often lose fine-grained spatial details. Existing multi-path hybrid models often face efficiency bottlenecks, making them impractical for clinical deployment.

MISSU addresses these challenges by proposing a unified framework that models both global context and local detail during training while remaining lightweight during inference.

Methodology

The MISSU architecture follows an encoder-decoder structure integrated with a Transformer bottleneck and a self-distillation path.

Figure 1: The MISSU architecture overview, featuring a hybrid encoder, Transformer layers, and a self-distillation mechanism.

Hybrid Encoder and Transformer Block

The model utilizes a 3D CNN encoder to extract hierarchical local features. These features are subsequently tokenized and processed by a Transformer block to capture global, long-range correlations across the 3D volume. This hybrid approach ensures that both fine-grained textures and broad structural relationships are preserved.

Multi-Scale Fusion and Self-Distillation

The "secret sauce" of MISSU is the Multi-Scale Fusion (MSF) block. Acting as a teacher module, it employs parallel atrous convolutions with varying dilation rates to capture rich, multi-scale information, particularly around complex organ boundaries. During training, a distillation loss ($L_{sd}$) encourages the main CNN encoder to mimic these high-quality features.

Crucially, the MSF block is discarded during inference. The knowledge it provides is distilled into the encoder's weights, allowing the model to maintain high accuracy without the added computational overhead of the teacher module at runtime.

Experiments and Results

MISSU was evaluated on the BraTS 2019 (brain tumor) and CHAOS (liver) datasets, demonstrating superior performance across multiple metrics.

Performance Benchmarks

On BraTS 2019, MISSU achieved state-of-the-art Dice scores of 89.98% for the Whole Tumor (WT), 85.77% for the Tumor Core (TC), and 80.14% for the Enhanced Tumor (ET). On the CHAOS dataset, it reached a 96.78% Dice score for liver segmentation.

Efficiency and Qualitative Analysis

Despite its performance, MISSU is highly efficient, utilizing only 10.50M parameters and 132.11 GFLOPs—significantly lower than standard 3D UNets or competing TransUNet variants. Qualitative results demonstrate MISSU's ability to accurately delineate irregular lesion boundaries compared to other methods.

Figure 6: Qualitative comparison of MISSU against other state-of-the-art methods on the BraTS 2019 and CHAOS datasets.

Conclusion

MISSU demonstrates that implicit feature fusion via self-distillation is a powerful strategy for 3D medical image segmentation. By shifting architectural complexity to the training phase, the model achieves a superior balance between accuracy and efficiency. Future work may involve extending this framework to other imaging modalities and integrating non-image clinical metadata to further enhance diagnostic precision.

Summary updated

BibTeX
@article{wang2023missu,
  author = {Nan Wang and Shaohui Lin and Xiaoxiao Li and Ke Li and Yunhang Shen and Yue Gao and Lizhuang Ma},
  title = {{MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet}},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2023},
  url = {https://openreview.net/forum?id=Ngc7RK0tmE},
}
Figure