DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation

Qiuyu Tian, Haoliang Sun, Yunshan Wang, Yinghuan Shi, Yilong YinarXiv 2026 · arXiv

TL;DR

DeferredSeg introduces a pixel-wise "learning-to-defer" framework for medical image segmentation, allowing an AI model to dynamically hand over ambiguous regions or boundaries to human experts while processing easy pixels automatically. By optimizing routing across single or multiple experts with spatial-coherence and load-balancing constraints, it significantly boosts system-level accuracy while minimizing expert workload to just a small fraction of the image.

What Problem Does This Paper Address?

Deep learning models excel at medical image segmentation but suffer from unreliable confidence scores, often showing overconfidence or underconfidence in ambiguous regions like fuzzy boundaries, low-contrast tissues, or small lesions. In high-stakes clinical settings, an incorrect prediction delivered with high certainty can mislead clinicians and cause severe consequences.

While "Learning to Defer" (L2D) paradigms exist to triage difficult cases to human experts, traditional L2D operates at the instance level (rejecting an entire image). In dense prediction tasks like segmentation, this is highly inefficient. A model might be correct for 95% of the image and struggle only at a specific boundary. Forcing a clinician to manually correct or inspect the entire image defeats the purpose of automation. DeferredSeg addresses this gap by formulating a region-selective, pixel-wise collaboration strategy.

Methods

The authors propose DeferredSeg, a backbone-compatible framework that extends a base segmentor (such as MedSAM or CENet) with pixel-level routing capabilities.

The architecture introduces several novel components to achieve this:

  1. Output Channel Extension: The base segmentor's output is augmented with $J+1$ routing channels. Channel 0 represents keeping the model's prediction, while channels $1$ to $J$ correspond to specific expert branches.
  2. Deferral Collaboration (DC) Loss: A pixel-wise surrogate loss that supervises the routing function, encouraging pixels to be routed to whichever branch (model or specific expert) is locally most reliable.
  3. Aggregated Deferral Predictor & Spatial-Coherence (SC) Loss: Modern Vision Transformer (ViT) decoders lack strong inductive biases for local spatial relationships, which can lead to fragmented, noisy deferral maps on small datasets. To fix this, an auxiliary CNN-based predictor generates a smooth pseudo-supervision mask, and an SC loss enforces spatial consistency.
  4. Load-Balancing (LB) Penalty: In multi-expert scenarios, this penalty prevents "routing collapse" (where the model over-relies on one expert and ignores others) by enforcing upper and lower bounds on expert utilization.
Figure 1: Overview of DeferredSeg. A deferral predictor produces a pixel-wise deferral map that triages image regions between human-expert branches and the base segmentation model, enabling spatially adap- tive Human–AI collaboration for deferral.

Datasets

The framework was evaluated across two distinct expert settings using six datasets:

  • Synthetic-Expert Benchmarks: PROMISE12 (Prostate MRI), LiTS (Liver/Tumor CT), and AMOS22 (Abdominal multi-organ CT/MRI). Here, expert annotations were simulated with controlled, region-specific error profiles to test how well the model learns to exploit specific expert strengths.
  • Offline Clinician-Annotation Benchmarks: Chaksu (Optic Disc/Cup Fundus images), QUBIQ (Brain/Prostate MRI), and CURVAS (Kidney/Liver CT). These utilized pre-existing annotations from multiple real clinicians to capture natural inter-annotator variability.

Key Insights and Ideas

  • Dense Triage: The core conceptual shift is treating deferral as a dense, localized decision. The AI handles clear, homogeneous regions, while humans focus strictly on tricky boundaries.
  • Fixing ViT's Spatial Blindspots with CNNs: The authors realized that ViT-based decoders struggle to learn clean, continuous deferral regions on small datasets. Complementing the ViT with a simple auxiliary CNN head to enforce spatial coherence via pseudo-labels dramatically stabilizes the routing.
  • Expert Complementarity Over Absolute Skill: In multi-expert tests, adding an expert who is globally mediocre but possesses highly specialized regional strengths (e.g., great at boundaries but poor at foreground) yields better system performance than adding multiple globally uniform experts. Multi-expert deferral benefits from complementary regional strengths rather than simply increasing the number of experts.

Key Results

  • Significant Performance Gains: DeferredSeg consistently outperformed fully automated baselines (nnU-Net v2, CENet, MedSAM) and patch-based correction baselines (PatchCorr). For instance, on the challenging LiTS Tumor dataset, it improved the Dice Similarity Coefficient (DSC) from 59.20% (MedSAM fine-tuned) to 91.01% (with 1 expert).
  • Low Expert Workload: The system achieved these gains while deferring only a tiny fraction of pixels. Across most datasets, the total deferral ratio was under 5% (e.g., 1.57% for LiTS Liver, 0.55% for Spleen), proving that clinicians only need to review highly localized regions.
  • Backbone Agnostic: Integrating DeferredSeg into CENet yielded similar statistically significant improvements, proving the framework generalizes across different network architectures.

Limitations and Future Work

  • Data-Hungry Training: Although it saves expert time during inference, training the routing function still relies on previously annotated multi-expert data, which can be costly and difficult to acquire.
  • Offline Evaluation: The current study simulates the human-in-the-loop workflow using offline annotations. Real-world clinical deployment will require prospective user studies to evaluate actual interaction efficiency, UI/UX design (e.g., freehand canvas tools), and the true cognitive load on clinicians.

Summary updated

Abstract

Segmentation models based on deep neural networks demonstrate strong generalization for medical image segmentation. However, they often exhibit overconfidence or underconfidence, leading to unreliable confidence scores for segmentation masks, especially in ambiguous regions. This undermines the trustworthiness required for clinical deployment. Motivated by the learning-to-defer (L2D) paradigm, we introduce DeferredSeg, a deferral-aware segmentation framework, i.e., a Human--AI collaboration system that determines whether to defer predictions to human experts in specific regions. DeferredSeg extends the base segmentor with an aggregated deferral predictor and additional routing channels that dynamically route each pixel to either the base segmentor or a human expert. To train this routing efficiently, we introduce a pixel-wise surrogate collaboration loss that supervises deferral decisions. In addition, to preserve spatial coherence within deferral regions, we propose a spatial-coherence loss that enforces smooth deferral masks, thereby enhancing reliability. Beyond single-expert deferral, we further extend the framework to a multi-expert setting by introducing multiple discrepancy experts for collaborative decision-making. To prevent overloading or underutilizing individual experts, we further design a load-balancing penalty that evenly distributes workload across expert branches. We evaluate DeferredSeg on three challenging medical datasets using MedSAM and CENet as the base segmentor for fair comparison. Experimental results show that DeferredSeg consistently outperforms the baseline, demonstrating its effectiveness for trustworthy dense medical segmentation. Moreover, the proposed framework is model-agnostic and can be readily applied to other segmentation architectures.
BibTeX
@misc{tian2026deferredseg,
  author = {Qiuyu Tian and Haoliang Sun and Yunshan Wang and Yinghuan Shi and Yilong Yin},
  title = {{DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation}},
  year = {2026},
  eprint = {2604.12411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.12411},
}
Figure