TL;DR
DeferredSeg introduces a pixel-wise "learning-to-defer" framework for medical image segmentation, allowing an AI model to dynamically hand over ambiguous regions or boundaries to human experts while processing easy pixels automatically. By optimizing routing across single or multiple experts with spatial-coherence and load-balancing constraints, it significantly boosts system-level accuracy while minimizing expert workload to just a small fraction of the image.
What Problem Does This Paper Address?
Deep learning models excel at medical image segmentation but suffer from unreliable confidence scores, often showing overconfidence or underconfidence in ambiguous regions like fuzzy boundaries, low-contrast tissues, or small lesions. In high-stakes clinical settings, an incorrect prediction delivered with high certainty can mislead clinicians and cause severe consequences.
While "Learning to Defer" (L2D) paradigms exist to triage difficult cases to human experts, traditional L2D operates at the instance level (rejecting an entire image). In dense prediction tasks like segmentation, this is highly inefficient. A model might be correct for 95% of the image and struggle only at a specific boundary. Forcing a clinician to manually correct or inspect the entire image defeats the purpose of automation. DeferredSeg addresses this gap by formulating a region-selective, pixel-wise collaboration strategy.
Methods
The authors propose DeferredSeg, a backbone-compatible framework that extends a base segmentor (such as MedSAM or CENet) with pixel-level routing capabilities.
The architecture introduces several novel components to achieve this:
- Output Channel Extension: The base segmentor's output is augmented with $J+1$ routing channels. Channel 0 represents keeping the model's prediction, while channels $1$ to $J$ correspond to specific expert branches.
- Deferral Collaboration (DC) Loss: A pixel-wise surrogate loss that supervises the routing function, encouraging pixels to be routed to whichever branch (model or specific expert) is locally most reliable.
- Aggregated Deferral Predictor & Spatial-Coherence (SC) Loss: Modern Vision Transformer (ViT) decoders lack strong inductive biases for local spatial relationships, which can lead to fragmented, noisy deferral maps on small datasets. To fix this, an auxiliary CNN-based predictor generates a smooth pseudo-supervision mask, and an SC loss enforces spatial consistency.
- Load-Balancing (LB) Penalty: In multi-expert scenarios, this penalty prevents "routing collapse" (where the model over-relies on one expert and ignores others) by enforcing upper and lower bounds on expert utilization.
Datasets
The framework was evaluated across two distinct expert settings using six datasets:
- Synthetic-Expert Benchmarks: PROMISE12 (Prostate MRI), LiTS (Liver/Tumor CT), and AMOS22 (Abdominal multi-organ CT/MRI). Here, expert annotations were simulated with controlled, region-specific error profiles to test how well the model learns to exploit specific expert strengths.
- Offline Clinician-Annotation Benchmarks: Chaksu (Optic Disc/Cup Fundus images), QUBIQ (Brain/Prostate MRI), and CURVAS (Kidney/Liver CT). These utilized pre-existing annotations from multiple real clinicians to capture natural inter-annotator variability.
Key Insights and Ideas
- Dense Triage: The core conceptual shift is treating deferral as a dense, localized decision. The AI handles clear, homogeneous regions, while humans focus strictly on tricky boundaries.
- Fixing ViT's Spatial Blindspots with CNNs: The authors realized that ViT-based decoders struggle to learn clean, continuous deferral regions on small datasets. Complementing the ViT with a simple auxiliary CNN head to enforce spatial coherence via pseudo-labels dramatically stabilizes the routing.
- Expert Complementarity Over Absolute Skill: In multi-expert tests, adding an expert who is globally mediocre but possesses highly specialized regional strengths (e.g., great at boundaries but poor at foreground) yields better system performance than adding multiple globally uniform experts. Multi-expert deferral benefits from complementary regional strengths rather than simply increasing the number of experts.
Key Results
- Significant Performance Gains: DeferredSeg consistently outperformed fully automated baselines (nnU-Net v2, CENet, MedSAM) and patch-based correction baselines (PatchCorr). For instance, on the challenging LiTS Tumor dataset, it improved the Dice Similarity Coefficient (DSC) from 59.20% (MedSAM fine-tuned) to 91.01% (with 1 expert).
- Low Expert Workload: The system achieved these gains while deferring only a tiny fraction of pixels. Across most datasets, the total deferral ratio was under 5% (e.g., 1.57% for LiTS Liver, 0.55% for Spleen), proving that clinicians only need to review highly localized regions.
- Backbone Agnostic: Integrating DeferredSeg into CENet yielded similar statistically significant improvements, proving the framework generalizes across different network architectures.
Limitations and Future Work
- Data-Hungry Training: Although it saves expert time during inference, training the routing function still relies on previously annotated multi-expert data, which can be costly and difficult to acquire.
- Offline Evaluation: The current study simulates the human-in-the-loop workflow using offline annotations. Real-world clinical deployment will require prospective user studies to evaluate actual interaction efficiency, UI/UX design (e.g., freehand canvas tools), and the true cognitive load on clinicians.
Summary updated