Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation

Zichen Luo, Xinshan Zhu, Lan Zhang, Biao SunAAAI 2025 · DOI

TL;DR

This paper introduces the Task-Adaptive Mixture of Skip Connections (TA-MOSC) module for U-Net, a popular medical image segmentation architecture. By treating skip connections as a task allocation problem using a Mixture of Experts (MoE) approach, TA-MOSC dynamically routes relevant encoder features to decoder stages, significantly improving segmentation accuracy and generalization across diverse medical imaging tasks with minimal added complexity.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis and treatment planning. The U-Net architecture, with its encoder-decoder structure and skip connections, has been a cornerstone in this field. Skip connections are vital for recovering spatial information lost during downsampling in the encoder. However, the original U-Net's fixed, one-to-one skip connection mechanism struggles to address the "semantic gap" between encoder and decoder features. This gap becomes more pronounced with diverse datasets and tasks, as different decoder stages may require varying types and amounts of information from the encoder. Existing U-Net variants often improve feature fusion but still rely on static skip connection strategies, limiting their adaptability to task-specific semantic needs and hindering generalization.

Methods

The core innovation is the Task-Adaptive Mixture of Skip Connections (TA-MOSC) module, inspired by the Mixture of Experts (MoE) framework. Instead of a direct, fixed connection, TA-MOSC treats the selection of encoder features for each decoder stage as a task allocation problem.

Here's how it works:

  • Feature Aggregation Stripe: Encoder features from multiple levels are concatenated and then reduced in dimensionality to form a unified representation.
  • Routing Phase (RP): A "router bank" dynamically learns to assign weights to different "expert" convolutional sub-networks. Each expert is specialized in extracting specific types of features. The router determines which combination of experts is most relevant for a given decoder stage and task.
  • Expert Modules: These are lightweight convolutional sub-networks designed to transform input features. The paper uses a sparse selection strategy (Top-K) to activate only a subset of experts, reducing computational overhead.
  • Docker Module: This module shapes and processes the output from the selected experts before feeding it as a skip connection to the corresponding decoder stage.
  • Balanced Expert Utilization (BEU): To prevent some experts from being underutilized or overworked, the authors introduce an "Expert Variance (EV) Loss." This loss encourages a more balanced distribution of tasks among experts, ensuring all are trained effectively. They also implement a mechanism to handle "unused experts" by processing a random sample of data through them.

The TA-MOSC module is integrated into the U-Net architecture, creating the proposed UTANet.

Datasets

The paper evaluates UTANet on four publicly available medical image segmentation datasets:

  • GlaS: A dataset for gland segmentation.
  • MoNuSeg: A dataset for nucleus segmentation.
  • Synapse: A dataset for multi-organ segmentation from abdominal CT scans.
  • ISIC16: A dataset for skin lesion segmentation.

These datasets represent a range of scales and complexities, allowing for a comprehensive evaluation of the model's performance and generalization capabilities.

Key Insights and Ideas

  • Skip Connections are Not One-Size-Fits-All: The paper's initial experiments (Figure 2) reveal that the optimal ordering of skip connections varies significantly across different datasets, suggesting that decoders have task-specific preferences for encoder information.
  • MoE for Adaptive Feature Routing: Applying the MoE paradigm to skip connections allows for dynamic, task-adaptive routing of features. This is a novel approach to bridging the semantic gap in U-Net.
  • Sparsity and Efficiency: Using sparse expert activation (Top-K) and lightweight convolutional experts keeps the computational overhead minimal, making the TA-MOSC module efficient.
  • Balanced Training is Crucial: The BEU mechanism, particularly the EV loss, is essential for ensuring that all specialized experts are effectively trained and contribute to the model's performance, preventing the model from relying too heavily on a few experts.

Key Results

  • State-of-the-Art Performance: UTANet consistently outperforms several strong baseline models (including U-Net, UNet++, Attention U-Net, TransUNet, and UDTransNet) across all four datasets.
  • Significant Dice Score Gains: The paper reports substantial improvements in Dice scores, for example:
  • GlaS: +2.65%
    • MoNuSeg: +6.55%
    • Synapse: +3.82%
    • ISIC16: +1.35%
  • Improved Stability and Robustness: The model demonstrates high stability, evidenced by small standard deviations in cross-validation results.
  • Qualitative Improvements: Visualizations show UTANet capturing more detailed information, particularly in challenging regions, compared to other models.

Limitations and Future Work

  • Limitations: While the TA-MOSC module adds minimal parameters, the paper doesn't extensively detail the computational cost increase compared to a standard U-Net. The "unused experts handling" mechanism is a heuristic approach to ensure expert training.
  • Future Work: The authors suggest optimizing the computational efficiency of the TA-MOSC module for better scalability and real-time applications. They also propose extending the approach to other dense prediction tasks beyond medical image segmentation, such as natural image segmentation and object detection.

Summary updated

BibTeX
@inproceedings{luo2025rethinking,
  author = {Zichen Luo and Xinshan Zhu and Lan Zhang and Biao Sun},
  title = {{Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32627},
}
Figure