Simple is what you need for efficient and accurate medical image segmentation

Xiang Yu, Yayan Chen, Guannan He, Qing Zeng, Yue Qin, Meiling Liang, Dandan Luo, Yimei Liao, Zeyu Ren, Kang Cheng, Delong Yang, Bocheng Liang, Bin Pu, Ying Yuan, Shengli Li

Abstract

While modern segmentation models often prioritize performance over practicality, we advocate a design philosophy prioritizing simplicity and efficiency, and attempted high performance segmentation model design. This paper presents SimpleUNet, a scalable ultra-lightweight medical image segmentation model with three key innovations: (1) A partial feature selection mechanism in skip connections for redundancy reduction while enhancing segmentation performance; (2) A fixed-width architecture that prevents exponential parameter growth across network stages; (3) An adaptive feature fusion module achieving enhanced representation with minimal computational overhead. With a record-breaking 16 KB parameter configuration, SimpleUNet outperforms LBUNet and other lightweight benchmarks across multiple public datasets. The 0.67 MB variant achieves superior efficiency (8.60 GFLOPs) and accuracy, attaining a mean DSC/IoU of 85.76%/75.60% on multi-center breast lesion datasets, surpassing both U-Net and TransUNet. Evaluations on skin lesion datasets (ISIC 2017/2018: mDice 84.86%/88.77%) and endoscopic polyp segmentation (KVASIR-SEG: 86.46%/76.48% mDice/mIoU) confirm consistent dominance over state-of-the-art models. This work demonstrates that extreme model compression need not compromise performance, providing new insights for efficient and accurate medical image segmentation.

Introduction

In the field of medical AI, the prevailing trend has been toward increasingly complex architectures. Researchers frequently pursue incremental accuracy gains by integrating heavy attention modules, Transformers, or state-space models. While these "heavy" models perform well in controlled environments, they are often impractical for clinical deployment on resource-constrained hardware, such as mobile ultrasound machines or edge devices.

SimpleUNet addresses a fundamental inefficiency in the classic U-Net architecture: exponential parameter growth. In standard U-Nets, the number of channels doubles at every downsampling stage. Furthermore, the practice of concatenating all feature maps through skip connections introduces significant information redundancy, forcing the decoder to process unnecessary noise. SimpleUNet challenges this paradigm by prioritizing structural simplicity and efficiency.

Methods: The SimpleUNet Architecture

SimpleUNet strips back the complexity of modern models and introduces three elegant architectural changes designed to maximize performance per parameter.

The comprehensive architecture of SimpleUNet, highlighting the fixed-width design, partial feature selection, and adaptive feature fusion modules.
  1. Fixed-Width Architecture: Unlike the standard U-Net, which doubles its channel count at each downsampling step, SimpleUNet maintains a constant width throughout the encoder. This shifts parameter growth from exponential to linear, significantly reducing the model's footprint.
  2. Efficient Feature Selection: Instead of passing every feature map through skip connections, the model uses $1 \times 1$ convolutions to select only the most representative features (partial selection). This drastically reduces the parameters required in the decoding stage while filtering out redundant information.
  3. Adaptive Feature Fusion (AFF): To maintain accuracy despite the reduced parameter count, the authors introduced learnable channel-wise weights ($\alpha$ and $eta$). This acts as a cost-minimal attention mechanism, allowing the network to dynamically prioritize the most important information during the fusion of shortcut and deep features.

Experimental Setup and Datasets

The model was rigorously evaluated across seven public datasets covering three distinct clinical domains, demonstrating its versatility across different imaging modalities.

Representative medical images and corresponding ground truth masks from the diverse datasets used for evaluation, including breast lesions, skin lesions, and polyps.
  • Merged Breast Dataset (MBD): A curated collection of ~3,000 ultrasound images from BUSI, DatasetB, BrEaST, and BUS-BRA.
  • Skin Lesion Datasets: ISIC 2017 and ISIC 2018, featuring dermoscopic images.
  • Endoscopic Polyp Segmentation: The KVASIR-SEG dataset, containing gastrointestinal polyp images.

Key Results and Discussion

SimpleUNet demonstrates that extreme model compression does not necessitate a compromise in performance. Key findings include:

  • Record-Breaking Compactness: A 16 KB variant of SimpleUNet outperformed LBUNet, a popular lightweight benchmark, across multiple datasets.
  • Superior Efficiency: The 0.67 MB variant achieved a mean Dice Similarity Coefficient (mDSC) of 85.76% on breast lesion datasets, surpassing TransUNet, a heavy Transformer-based model that is roughly 140 times larger (93.23 MB).
  • Redundancy Reduction: A critical insight from this work is that concatenating only partial features (e.g., a 50% selection rate) often leads to better segmentation performance than using the full set. This suggests that standard U-Nets are often hindered by "noise" in their skip connections.
  • Scalability: The architecture is highly interpretable and scalable, allowing researchers to tune model width and selection rates to fit specific deployment environments, from tiny edge devices to robust server-side applications.

Conclusion

SimpleUNet proves that refining the basic structural logic of the U-Net can yield better results than adding "obscure" complex modules. By addressing exponential width growth and skip connection redundancy, SimpleUNet provides a new benchmark for efficient and accurate medical image segmentation. Future work will explore spatial reduction in initial stages and the potential benefits of deep supervision to further optimize this ultra-lightweight framework.

The code is publicly available at: https://github.com/Frankyu5666666/SimpleUNet

Summary updated

Abstract

While modern segmentation models often prioritize performance over practicality, we advocate a design philosophy prioritizing simplicity and efficiency, and attempted high performance segmentation model design. This paper presents SimpleUNet, a scalable ultra-lightweight medical image segmentation model with three key innovations: (1) A partial feature selection mechanism in skip connections for redundancy reduction while enhancing segmentation performance; (2) A fixed-width architecture that prevents exponential parameter growth across network stages; (3) An adaptive feature fusion module achieving enhanced representation with minimal computational overhead. With a record-breaking 16 KB parameter configuration, SimpleUNet outperforms LBUNet and other lightweight benchmarks across multiple public datasets. The 0.67 MB variant achieves superior efficiency (8.60 GFLOPs) and accuracy, attaining a mean DSC/IoU of 85.76%/75.60% on multi-center breast lesion datasets, surpassing both U-Net and TransUNet. Evaluations on skin lesion datasets (ISIC 2017/2018: mDice 84.86%/88.77%) and endoscopic polyp segmentation (KVASIR-SEG: 86.46%/76.48% mDice/mIoU) confirm consistent dominance over state-of-the-art models. This work demonstrates that extreme model compression need not compromise performance, providing new insights for efficient and accurate medical image segmentation. Codes can be found at https://github.com/Frankyu5666666/SimpleUNet.
BibTeX
@inproceedings{yu2025simple,
  author = {Xiang Yu and Yayan Chen and Guannan He and Qing Zeng and Yue Qin and Meiling Liang and Dandan Luo and Yimei Liao and Zeyu Ren and Kang Cheng and Delong Yang and Bocheng Liang and Bin Pu and Ying Yuan and Shengli Li},
  title = {{Simple is what you need for efficient and accurate medical image segmentation}},
  booktitle = {Expert},
  year = {2025},
  doi = {10.1016/j.eswa.2025.130687},
}
Figure