UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer

Haonan Wang, Peng Cao, Jiaqi Wang, Osmar R. Zaïane

Abstract

Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a proposed CTrans module in U-Net), from the channel perspective with attention mechanism. Specifically, the CTrans (Channel Transformer) module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity. Hence, the proposed connection consisting of the CCT and CCA is able to replace the original skip connection to solve the semantic gaps for an accurate automatic medical image segmentation. The experimental results suggest that our UCTransNet produces more precise segmentation performance and achieves consistent improvements over the state-of-the-art for semantic segmentation across different datasets and conventional architectures involving transformer or U-shaped framework.

Introduction: The Skip Connection Paradox

The U-Net architecture has long been considered the gold standard for medical image segmentation, primarily due to its skip connections that shuttle low-level spatial information from the encoder to the decoder. However, this design assumes that encoder and decoder features at the same resolution are semantically compatible.

In reality, a significant "semantic gap" exists. Encoder features are often noisy or too low-level, while decoder features are highly processed. Analysis reveals a "Skip Connection Paradox": on certain datasets like GlaS, a U-Net with no skip connections actually performs better than a standard one. This suggests that simple concatenation can introduce ambiguity that harms final segmentation accuracy.

Figure 3: Dice values across different skip connection settings. Note how 'None' outperforms 'All' on the GlaS dataset, highlighting the semantic gap issue.

Proposed Method: UCTransNet

To address these limitations, we propose UCTransNet, which replaces standard skip connections with the CTrans (Channel Transformer) module. This module shifts the focus from spatial-wise attention to channel-wise interactions to better bridge the semantic gap.

Figure 2: The UCTransNet architecture featuring the CCT (Channel-wise Cross Fusion Transformer) and CCA (Channel-wise Cross-Attention) modules.

Channel-wise Cross Fusion Transformer (CCT)

Unlike standard Vision Transformers (ViTs) that focus on spatial patches, CCT performs attention along the channel axis. It takes features from all four levels of the encoder simultaneously, allowing the model to learn long-range dependencies across different scales and channels. This "multi-scale fusion" ensures the decoder receives a rich, unified representation rather than a single-scale raw copy.

Channel-wise Cross-Attention (CCA)

Once multi-scale features are fused by the CCT, the CCA module acts as a filter. It uses the decoder's current features to "guide" the fusion, effectively masking out irrelevant information from the Transformer output and narrowing the semantic gap before final concatenation.

Experimental Results

UCTransNet was evaluated on three widely recognized medical imaging benchmarks: GlaS (gland segmentation), MoNuSeg (multi-organ nuclei segmentation), and Synapse (multi-organ abdominal CT).

  • GlaS: Achieved a 90.18% Dice score, a significant jump over vanilla U-Net (85.45%) and Swin-Unet (89.58%).
  • MoNuSeg: Reached 79.08% Dice, outperforming the baseline by nearly 3%.
  • Synapse: Showed strong competitive performance, particularly in reducing the Hausdorff Distance (HD), which measures boundary error.

Ablation studies confirmed that the synergy between CCT and CCA is essential; while using either provides a boost, their combination drives the state-of-the-art performance.

Conclusion

UCTransNet rethinks skip connections by treating encoder stages as a collaborative source of information. By leveraging channel-wise Transformers and cross-attention, it effectively solves the semantic gap problem. The model consistently outperforms both traditional U-shaped frameworks and recent spatial-Transformer hybrids, providing a more robust solution for accurate medical image segmentation.

Summary updated

Abstract

Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a proposed CTrans module in U-Net), from the channel perspective with attention mechanism. Specifically, the CTrans (Channel Transformer) module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity. Hence, the proposed connection consisting of the CCT and CCA is able to replace the original skip connection to solve the semantic gaps for an accurate automatic medical image segmentation. The experimental results suggest that our UCTransNet produces more precise segmentation performance and achieves consistent improvements over the state-of-the-art for semantic segmentation across different datasets and conventional architectures involving transformer or U-shaped framework. Code: https://github.com/McGregorWwww/UCTransNet.
BibTeX
@inproceedings{wang2022uctransnet,
  author = {Haonan Wang and Peng Cao and Jiaqi Wang and Osmar R. Zaïane},
  title = {{UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=1OZrDb2e8U},
}
Figure