U-Bench: A Comprehensive Understanding of U-Net through 100-Variant Benchmarking

Fenghe Tang, Chengqi Dong, Wenxin Ma, Zikang Xu, Heqin Zhu, Zihang Jiang, Rongsheng Wang, Yuhao Wang, Chenxu Wu, S Kevin ZhouSubmission to ICLR 2026 · Rejected

TL;DR

Over the past decade, hundreds of U-Net variants have claimed state-of-the-art segmentation performance, yet few have undergone rigorous statistical validation or efficiency testing. U-Bench establishes the largest standardized benchmark to date—evaluating 100 U-Net architectures across 28 datasets and 10 imaging modalities. The study reveals that while raw in-domain accuracy gains have largely plateaued (with over 80% of variants showing no statistically significant gain over baseline U-Net), modern variants provide genuine benefits in zero-shot generalization and efficiency trade-offs, captured here via a new deployment-focused metric called U-Score.

What Problem Does This Paper Address?

Since its debut in 2015, Ronneberger et al.’s U-Net has inspired thousands of architectural iterations across CNNs, Transformers, State-Space Models (e.g., Mamba), and RNN hybrids (e.g., RWKV). However, benchmarking practices in the medical segmentation community remain fragmented:

  • 73% of papers report gains without formal statistical hypothesis testing.
  • 84% ignore zero-shot or out-of-distribution evaluation, leaving cross-institution robustness unmeasured.
  • Computational efficiency (parameter size, FLOPs, and FPS) is frequently overlooked, leading to heavyweight models ill-suited for edge clinical hardware.

U-Bench bridges this gap with a standardized, reproducible 2D benchmark to evaluate whether 10 years of architectural innovation translates into statistically meaningful, clinically deployable improvements.

Methods

  1. Massive Cross-Architecture Evaluation: Implemented and standardized 100 open-source U-shaped variants spanning five architectural paradigms: CNNs, pure Vision Transformers, Mamba (SSMs), RWKV, and Hybrid architectures (combining CNN encoders/decoders with attention blocks).
  2. Standardized Training Recipe: All models were trained from official implementations under consistent preprocessing (256×256 standard resolution), uniform augmentation, unified loss formulations ($0.5 \times \text{BCE} + 1.0 \times \text{Dice}$), and identical hardware environments (NVIDIA H20 GPUs; over 6,000 total GPU hours).
  3. The U-Score Metric: To capture real-world deployability, the authors introduced U-Score, a harmonic-mean metric combining percentile-normalized accuracy ($a_i$ based on IoU) and an efficiency subscore ($\text{Eff}_i$) derived from model parameters ($p_i$), FLOPs ($g_i$), and inference speed in FPS ($s_i$): $$\text{U-Score}_i = \frac{2}{\frac{1}{a_i} + \frac{1}{\text{Eff}_i}}$$
  4. Model Advisor Agent: Trained an XGBoost pairwise ranking model (XGBRanker) using dataset-level geometric descriptors (foreground scale, boundary blur, shape regularity) and user-specified hardware constraints (storage, compute budget, target FPS) to automatically recommend the best model candidates for new tasks.

Datasets

The authors compiled 28 public 2D segmentation datasets spanning 10 medical imaging modalities:

  • Ultrasound: BUSI, BUS, BUSBRA, TNSCUI, TUCC
  • Dermoscopy: ISIC2018, PH2, SkinCancer
  • Endoscopy: Kvasir-SEG, CVC-300, CVC-ClinicDB, Robotool
  • Fundus Photography: CHASE, Stare, DRIVE, DCA
  • X-Ray: Covidquex, Montgomery, NIH-test
  • MRI: Promise, ACDC
  • CT: Synapse multi-organ
  • OCT: Cystoidfluid
  • Nuclear / Histology: DSB2018, Cell, Monusac, Tnbcnuclei, Glas

20 datasets were dedicated to in-domain supervised training and evaluation, while 8 datasets served exclusively as held-out targets for cross-dataset, zero-shot transfer within the same modality (e.g., training on Kvasir-SEG and zero-shot evaluating on CVC-300).

Key Insights and Ideas

  • In-domain gains are mostly noise: Over 80% of U-Net variants fail to achieve statistically significant improvements ($p < 0.05$) over the original 2015 vanilla U-Net on source domains. Reported IoU gains are often small fluctuations (1–2% on average).
  • The real breakthrough is generalization: In cross-center zero-shot evaluations, modern architectures significantly outperform vanilla U-Net (>3% mean gain; >50% of outperforming models reach statistical significance across 75% of modalities). Global context mechanisms (Transformers, RWKV, Mamba) improve out-of-distribution transfer substantially.
  • CNNs dominate practical deployment: While Hybrid models achieve top raw IoU by pairing local inductive priors with global attention, the U-Score leaderboard shifts heavily toward streamlined CNNs (occupying 7 of the top 10 spots) due to their superior throughput and parameter efficiency.
  • Emergence of RWKV: Recurrent-inspired RWKV models (like RWKV-UNet) deliver top-tier segmentation accuracy alongside lightweight linear-time inference, beating both pure Transformers and Mamba-based variants in accuracy–efficiency trade-offs.
  • Task-specific architectural fit: Global semantic modeling primarily helps lesion localization (Ultrasound, Endoscopy), whereas repetitive structural segmentation (retinal vessels, cell nuclei) relies heavily on fine local boundary delineation, where complex global models offer diminishing returns.

Key Results

  • In-Domain Top Performers (IoU): Led by RWKV-UNet (79.84% mean IoU), nnU-Net, and AURA-Net. Vanilla U-Net still places #25 out of 100.
  • Zero-Shot Top Performers (IoU): RWKV-UNet (70.94% mean IoU), G-CASCADE, and Swin-umamba. Vanilla U-Net drops to #72, confirming modern architectures generalize far better.
  • Efficiency Rankings (U-Score): LGMSNet, MBSNet, and CMUNeXt take the top in-domain spots. Vanilla U-Net ranks #64, underscoring that decade-long progress has yielded +33% average improvements in accuracy-to-compute efficiency.
  • Advisor Agent: Achieves strong ranking alignment with ground-truth model orderings on held-out tasks ($\text{NDCG@10} = 0.84\text{--}0.87$ across U-Score metrics), finding a top-10% performing model in 1–2 trials.

Limitations and Future Work

  • 2D Slice Focus: 3D volumetric datasets (e.g., CT, MRI) were evaluated primarily as 2D slices; full 3D context models were not benchmarked.
  • Clinical Integration: Metrics currently balance mathematical accuracy (IoU, Dice, HD95) with compute performance (FPS, FLOPs). Future work plans to incorporate clinician-in-the-loop qualitative scoring and clinical diagnostic tolerance limits into the evaluation criteria.

Summary updated

Abstract

Over the past decade, U-Net has been the dominant architecture in medical image segmentation, leading to the development of thousands of U-shaped variants. Despite its widespread adoption, there is still no comprehensive benchmark to systematically evaluate their performance and utility, largely because of insufficient statistical validation and limited consideration of efficiency and generalization across diverse datasets. To bridge this gap, we present U-Bench, the first large-scale, statistically rigorous 2D benchmark that evaluates 100 U-Net variants across 28 datasets and 10 imaging modalities. Our contributions are threefold: (1) Comprehensive Evaluation: U-Bench evaluates models along three key dimensions: statistical robustness, zero-shot generalization, and computational efficiency. We introduce a novel metric, U-Score, which jointly captures the performance-efficiency trade-off, offering a deployment-oriented perspective on model progress. (2) Systematic Analysis and Model Selection Guidance: We summarize key findings from the large-scale evaluation and systematically analyze the impact of dataset characteristics and architectural paradigms on model performance. Based on these insights, we propose a model advisor agent to guide researchers in selecting the most suitable models for specific datasets and tasks. (3) Public Availability: We provide all code, models, protocols, and weights, enabling the community to reproduce our results and extend the benchmark with future methods. In summary, U-Bench not only exposes gaps in previous evaluations but also establishes a foundation for fair, reproducible, and practically relevant benchmarking in the next decade of U-Net-based segmentation models. The weights, datasets, and results will be released after the acceptance.
BibTeX
@misc{tang2025u,
  author = {Fenghe Tang and Chengqi Dong and Wenxin Ma and Zikang Xu and Heqin Zhu and Zihang Jiang and Rongsheng Wang and Yuhao Wang and Chenxu Wu and S Kevin Zhou},
  title = {{U-Bench: A Comprehensive Understanding of U-Net through 100-Variant Benchmarking}},
  year = {2025},
  eprint = {2510.07041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07041},
}
Figure