Medical imaging datasets are now mostly built by having a model draft a segmentation and a clinician correct it. This works well: correcting drafts cut labeling time for 3D middle-ear scans from five hours to twenty minutes. But these pipelines treat the clinician's attention as free and constant. In practice reviewers tire. A draft that looks plausible tempts a tired reviewer to accept it without checking, and the volunteer radiologists these efforts depend on stop coming back once the work feels like a second shift. We propose to treat reviewer attention as the scarce, depleting resource it is. Each reviewer's reliability would be measured continuously during the session using a small number of planted test cases. Those measurements would then decide three things: which cases that reviewer sees, when the session should end, and how load is spread across the volunteer pool so that people keep contributing.
In two sentences: crowdsourcing has long used hidden "gold" questions and fatigue-aware task allocation to keep cheap workers accurate and engaged. Clinical model-assisted labeling has neither. We bring both to expert segmentation review, run it inside short reading-room sessions, and test whether it yields more correct labels per clinician-hour and better volunteer return rates than routing cases by model uncertainty alone.
Core Claim
The quantity that matters in clinician-in-the-loop refinement is the number of verified-correct voxels obtained per minute of expert time, accumulated across the weeks a volunteer stays active. Two hidden failure modes push this number down. The first is over-acceptance: a reviewer approves a draft that contains an error, and the rate of this rises with time in session. The second is attrition: reviewers stop showing up. Neither failure can be seen from the corrected labels themselves.
Our central technical move is the seeded-error probe. This is a case where the gold segmentation is already known and the system has deliberately corrupted the draft. Examples include adding a false lesion region over a vessel or the choroid plexus, which mimics the failure that BraTS 2021 reports automated models make, or trimming a boundary. Probes are mixed into the queue at a low rate. Whether the reviewer catches the planted error gives a running, per-person, time-varying estimate of their current reliability.
That estimate drives three decisions. It ends a session when the estimated miss rate rises. It routes the hardest, highest-uncertainty cases to reviewers early in their sessions, while they are fresh. It spreads hard cases across the pool so that no single volunteer absorbs them all. We claim this reliability signal is what makes the difference. Routing by model uncertainty alone, which every reviewer would suggest, cannot detect over-acceptance, because the model's uncertainty says nothing about the reviewer.
Where This Sits
The model-assisted labeling work supplies the basic loop that we reuse. Three studies all run the same seed-train, draft, correct, retrain cycle offline:
- The Dresden in vivo OCT dataset trains nnU-Net on 14 hand-labeled volumes and merges two raters with model output using STAPLE, an algorithm that estimates each rater's accuracy after the fact.
- Yue et al. seed a Res-UNet with 100 breast MRI cases to draft 900 more.
- BraTS 2021 has an nnU-Net/DeepMedic/DeepScan ensemble draft 2,040 cases for volunteer neuroradiologists, with senior sign-off.
None of the three measures how reliable a reviewer is while the review is happening, or whether reviewers keep participating. STAPLE comes closest, since it estimates rater performance. We differ in two ways. Our estimate is made online and changes over the course of a session. It is also used to control the review process rather than only to merge the finished labels.
The crowdsourcing retention work motivates the allocation side.
- Coupons as Monetary Incentives in Participatory Sensing adds a "loser's bonus" so that repeatedly unselected participants do not drift away. We borrow the logic of spreading burden to protect the pool. We do not borrow the auction, whose bidding imposes the kind of mental load clinicians will not accept.
- First learn then earn shows that per-person behavioral profiles beat preference-agnostic allocation by 25% to over 150% in single-task settings. That work is static and one-shot, and our reliability profile is dynamic.
- Dynamic Incentive Mechanism Design for COVID-19 Social Distancing learns point rewards with a Deep Q-Network, a reinforcement-learning controller, but notes that long-term reward fatigue was never validated in live use.
We deliberately do not use rewards as the lever. Points or micropayments are a poor fit for clinicians. The lever under the platform's control is what work it asks of whom, and when.
The clinical integration literature constrains the design.
- Wang et al.'s systematic review finds that tools must be "unremarkable," meaning they need no separate logins or context switches. It also finds that only 7 of 20 studies tested tools in real clinical settings.
- The GutGPT trial shows that perceived ease of use, which rose by a median of 0.6 versus 0.3, did not translate into adoption intention or into accuracy, which was about 92% in both arms.
Together these rule out designs that rely on long chat-style interaction or external tools. They also tell us that self-reported usability is not an acceptable outcome measure. Instead we measure who comes back and how many planted errors are missed.
Monitoring Deployed AI Systems in Health Care treats clinician audit labor as untracked overhead and uses deliberately slow alert thresholds to avoid operator fatigue. Our reliability estimates could feed that kind of monitoring, but we do not claim to solve post-deployment governance.
Research Program
We would build a review service that plugs into a standard DICOM viewer and presents work in short slots of 10 to 20 minutes. A slot length of this kind fits between clinical reads.
The draft model is nnU-Net. Its per-voxel uncertainty comes from the cyclical-SGD checkpoint ensembles of Efficient Bayesian Uncertainty Estimation for nnU-Net. That paper reports better calibration and Dice than MC-Dropout and temperature scaling on ACDC and M&M, though the file quotes no raw numbers.
Each reviewer has a reliability state, modeled as the probability of missing a planted error. This probability is estimated from probe outcomes as a function of time in session, case difficulty, and draft quality. It is fitted hierarchically, so that sparse per-person data borrow strength from the pool.
The routing policy uses this state as follows. When a reviewer opens a slot, the policy assigns cases to maximize expected corrected-error mass. That quantity is the model's uncertainty-weighted expected error on the case, multiplied by the reviewer's current detection probability. The slot ends early once detection probability falls below a threshold. A pool-level constraint caps each person's share of hard cases over a rolling week, which is our analogue of the loser's bonus.
Probes are drawn from cases whose gold labels already exist. Corruptions are sampled from the error types that the real model makes on each dataset. The probe rate is itself a parameter we study, because probes cost reviewer time.
Baselines
The do-nothing reference is the protocol used by BraTS 2021 and DIOME. Cases arrive in random order, every case shows the model draft, sessions have no length cap, and no reviewer reliability is tracked. It is run with the same draft model, the same case pool, and the same total reviewer-minutes as every other arm.
The strongest existing method is uncertainty-sampling active learning with a fixed session cap. Here the most uncertain cases, scored by the same checkpoint ensemble, go to whichever reviewer is available, and sessions stop at a fixed minute count tuned on pilot data. This is the natural engineering answer, and it captures most of what model-side intelligence can offer. It is run with identical uncertainty estimates and the same cap budget, and the cap length is tuned with the same number of pilot sessions our policy gets.
As the reference for automated segmentation quality, the nnU-Net ensemble that won BraTS 2020 reports test Dice of 88.95 (whole tumor), 85.06 (tumor core), and 82.03 (enhancing tumor). For refinement efficiency, DIOME reports a 15x time reduction and Yue et al. report Dice 0.89 with a 60-fold speedup. No reported result was found on Lacuna for uncertainty-routed clinician refinement measured as residual label error per reviewer-minute, so the strongest baseline's number on our metric will be established in this study.
Two ablations isolate the key ingredient. The first adds probes but uses them only for STAPLE-style weighting after the fact, so reliability is measured but does not control stopping or routing. The second uses probes for stopping but not for pool balancing.
Beating the strongest baseline means two things. First, at equal total reviewer-minutes, the full policy must yield at least 20% fewer residual errors in accepted labels, measured against gold. Second, the downstream model retrained on those labels must show no worse Dice and HD95, the 95th-percentile boundary distance. In the prospective study, a higher session-return rate counts as a secondary win, but not a substitute for the label-quality result.
Why not just…?
Why not just route by model uncertainty and cap sessions at a fixed length? This is the real threat. If reviewer reliability declines at roughly the same rate for everyone, and that decline is predictable from elapsed time alone, a well-tuned fixed cap will match our adaptive stopping. The probes would then be wasted minutes. The first ablation and the strongest baseline exist to settle exactly this. If miss rates on probes do not vary meaningfully across reviewers and sessions beyond what elapsed time predicts, we will report that the simpler policy suffices and drop adaptive stopping. In that case the contribution would reduce to the probe-based measurement of over-acceptance, which is still useful as an audit tool.
Why not just hire paid annotators on flat contracts? Paying people does not make them stop tiring. Over-acceptance of plausible drafts is a property of verification work, not of motivation, and Monitoring Deployed AI Systems in Health Care and SlideChat both flag reviewer fatigue and over-reliance as error sources. Cheaper trainee labor is also not a safe substitute: in DIOME, nnU-Net outperformed the medical-student rater. Our method applies to paid annotators unchanged. Only the pool-balancing component is specific to volunteers.
Why not just skip the humans and use the model's output? On distribution-shifted data, segmentation models fail with high confidence. Examples are the cross-vendor M&M setting in Zhao et al. and the vessel and choroid-plexus false positives in BraTS 2021. Our M&M experiment includes an "accept all drafts" arm, so the gap between automated and human-reviewed labels on shifted data is measured directly rather than assumed.
Proposed Experiments
The retrospective stage is used only to engineer the policy, not to prove the claim. On BraTS 2021, we hold out a gold test split and treat the remaining gold labels as hidden truth. On M&M, we train on vendor A and refine on vendor B, where uncertainty matters most. Reviewer behavior is simulated with miss-rate curves fitted to a short pilot of real probe outcomes. The goals are to set probe rate, stopping threshold, and balancing cap, and to check that the policy does not degrade when the simulated fatigue curve is flat.
The prospective stage is the decisive test. We will recruit 12 to 20 radiologists and senior residents as volunteers for eight weeks. They review public BraTS 2021 and M&M cases in the embedded viewer, so no patient care is involved. Each volunteer is randomized in a crossover design between the full policy and the strongest baseline, in two four-week blocks. Outcomes are:
- residual errors in accepted labels against gold;
- probe miss rate as a function of minutes into the session;
- minutes per case;
- Dice and HD95 of an nnU-Net retrained on each arm's labels;
- session-return rate, meaning the fraction of offered slots a volunteer chooses to take.
With this sample, retention effects will be underpowered. We pre-register them as exploratory and treat label quality per minute as primary.
Two checks would falsify the premise. If probe miss rates are flat over the course of sessions and across reviewers, over-acceptance is not a real lever and the core claim fails. If the full policy does not beat uncertainty sampling with a tuned fixed cap on residual error at equal reviewer-minutes, the reliability signal is not doing work, and we will say so.
References
- The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
- The Dresden in vivo OCT dataset for automatic middle ear segmentation
- Deep learning-based automatic segmentation for size and volumetric measurement of breast cancer on magnetic resonance imaging
- Coupons as Monetary Incentives in Participatory Sensing
- First learn then earn: optimizing mobile crowdsensing campaigns through data-driven user profiling
- Dynamic Incentive Mechanism Design for COVID-19 Social Distancing
- Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review
- Usability and adoption in a randomized trial of GutGPT a GenAI tool for gastrointestinal bleeding
- Monitoring Deployed AI Systems in Health Care.
- Efficient Bayesian Uncertainty Estimation for nnU-Net
- nnU-Net for Brain Tumor Segmentation
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding