When two radiologists outline the same lung nodule, they often draw different boundaries, and neither is wrong. Interactive segmentation tools, where a model proposes a mask and the clinician fixes it with a few clicks, ignore this. They are trained and evaluated as if one true mask exists. On ambiguous tissue this is costly. A tool that proposes the "average" boundary forces every clinician to spend clicks pulling it toward their own style. A tool that reads each click only as a local correction misses what the click says about the clinician's overall style. We propose to train the mask-proposing model to minimize a different quantity: the expected number of clicks a clinician, whose style is unknown, needs before accepting the mask as their own. The model makes this decision one step at a time, and its objective counts steps, so it cannot be differentiated directly.
Our method borrows from modular imitation learning. This line of work first learns a fixed scoring function from demonstrations and only afterwards trains a policy against it, instead of training both together. Concretely, we pretrain and then freeze a critic that predicts how many clicks remain given the click history and a candidate mask. The critic is trained offline on simulated editing sessions from every annotator in multi-rater datasets. A mask-update policy is then extracted against this frozen critic. The claim is specific and falsifiable. On datasets with several annotators per image, and when we test on annotators held out from training, a policy trained against a frozen effort critic should reach each clinician's own acceptance threshold in fewer clicks than two alternatives. The first is the same network trained by ordinary per-step supervision toward a randomly drawn rater's mask. The second is a critic trained jointly with the policy.
Core Claim
Two lines of work each cover half of this problem. Multi-rater segmentation models already represent annotator style as a latent variable. Pionono gives each rater a Gaussian in an 8-dimensional space, where the mean encodes systematic bias and the variance encodes the rater's inconsistency with themselves. Liu et al.'s variational model samples plausible alternative boundaries from a 6-dimensional latent. MoSE matches a mixture of decoders to the empirical set of rater masks with an optimal-transport loss. All three are single-pass generators and never reason about an interaction sequence.
Interactive systems do reason about sequences, but under a single ground truth. PseudoClick imitates human clicking by placing a point at the centre of the largest error region against one target mask, and its conclusion explicitly leaves reinforcement learning over click sequences as future work. MedUHIP is the closest existing system. It adapts a sampling space of plausible masks to each clinician's clicks on REFUGE2, LIDC-IDRI and QUBIQ, but it is trained with per-interaction supervision and never optimizes total effort.
Our contribution sits between these. We treat the unknown rater as a hidden variable in a sequential decision problem, and we optimize the clicks-to-acceptance objective with a frozen, offline-trained critic. Let $x$ be an image and $r$ a rater drawn from the training rater distribution $p(r)$, with mask $y_r$. Let $\pi$ be the mask-update policy, which sees only the image and the click history $h_t$ and outputs a mask $m_t$. The objective is
$$ J(\pi) = \mathbb{E}_{x,\; r \sim p(r)}\big[\, T_\tau(\pi, x, r) \,\big], \qquad T_\tau = \min\{t : \mathrm{Dice}(m_t, y_r) \ge \tau\}, $$
where $\tau$ is the acceptance threshold (80, 85 or 90 percent Dice) and a click cap bounds $T_\tau$.
This objective differs from minimizing per-step Dice to a randomly sampled rater. The first mask that minimizes expected clicks need not be the mean mask. The best response to a click may be to commit to a whole style rather than to fix the clicked region. Whether that difference is large in practice is exactly what the experiments test.
Research Program
The first stage builds per-rater click simulators and an offline trajectory dataset. For every image and every training rater, we simulate editing sessions with PseudoClick's rule: click the centre of the largest false-positive or false-negative component. The difference is that errors are measured against that rater's own mask, not a consensus.
To avoid tying the critic to one behaviour policy, we generate sessions from several base mask-updaters. These are a MedUHIP-style latent-adaptation model, a PseudoClick model trained on majority vote, and randomized mixtures of rater samples from a Pionono-style generator. Covering this range matters because of the main known weakness of decoupled critics, noted for DVPO: a frozen critic gives wrong credit once the policy wanders into states the offline data never covered.
The second stage trains the critic $Q_\phi(x, h_t, m)$. It predicts the expected remaining clicks when mask $m$ is shown after history $h_t$, with the rater hidden. Training uses temporal-difference learning on the offline sessions, following DVPO's recipe of pretraining a value model and then freezing it. The critic is conditioned on the same image encoder as the policy.
Freezing is the ingredient taken from modular imitation learning. Random Expert Distillation and Energy-Based Imitation Learning both report that replacing a co-adapting adversary with a fixed objective removes large training instabilities. On Walker2d, RED reports a standard deviation of about 20 against GAIL's roughly 2848. We expect the same logic to apply when the "adversary" is a critic chasing a policy whose masks keep changing.
The third stage extracts the policy. The policy network is initialized from the supervised multi-rater backbone. We then fine-tune it with advantage-weighted regression against the frozen critic: the policy imitates offline mask proposals, weighted by how much they lower predicted remaining clicks. We also run a small number of PPO steps with a trust-region penalty. The advantage-weighted variant keeps the policy near the offline data, as OTR does when it pairs fixed rewards with IQL.
Finally, we audit the critic. We roll out the final policy against held-out raters, compare the critic's predicted remaining clicks with the true count, and report the calibration gap by interaction step. If the gap grows sharply as the policy moves away from the data, the decoupling has failed in the way DVPO warns about, and we will report that.
Baselines
The naive reference is a single deterministic interactive model trained on the majority-vote mask with PseudoClick-style simulated clicks, evaluated against each rater's own mask. It represents current practice: fuse labels, then build the tool. DiFF already shows that majority voting discards information relevant to diagnosis, and we expect this baseline to need the most clicks on ambiguous structures such as the optic cup.
The strongest existing method is MedUHIP, which adapts its sampling space to each clinician's clicks and was evaluated on the same multi-rater datasets. Its paper reports reaching 80% Dice on REFUGE2 in 2.72 interactions on average, against 3.65 for SAM. It also reports an average Dice of 89.68% after three interactions across datasets and roughly a 20% improvement over prior methods on LIDC. Those numbers come from MedUHIP's own protocol, so we will reimplement it inside our shared codebase. The shared elements are the same backbone, the same per-rater click simulator, the same held-out raters, and the same hyperparameter search budget in GPU-hours. We will also report our run next to the published 2.72 figure so any protocol mismatch is visible. Beating it means a statistically reliable reduction in mean clicks to 80% and 85% Dice against held-out raters' masks on REFUGE2 and LIDC-IDRI, with no loss in Dice after three clicks. The significance test is a paired bootstrap over images.
The ablation that isolates the key ingredient is the same policy network with the same data and compute, trained by per-step supervision alone. At each step, the target is the hidden rater's mask given the click history. This is the natural way to train a latent-inference interactive model, and it differs from our method only in the effort-aware objective.
A second ablation keeps the effort objective but trains the critic online, jointly with the policy. It tests whether freezing matters or whether any actor-critic method would do. For single-pass comparison, and to check that the policy's first mask is still a reasonable distribution match, we also report Generalized Energy Distance against MoSE on LIDC-IDRI. MoSE's paper reports reduced Generalized Energy Distance relative to PHiSeg and Probabilistic U-Net, but the file gives no absolute value on Lacuna, so we will measure it ourselves.
Why not just…?
Why not just infer the rater's latent style from the clicks and output that rater's mask, with no reinforcement learning? This is the most serious threat, and it is essentially the supervised ablation plus MedUHIP. If posterior inference over a Pionono-style latent already makes each step's mask the best guess of the rater's style, then minimizing per-step error and minimizing total clicks may nearly coincide. The two objectives should diverge in specific places: at the first proposal, where the mean mask can sit between rater modes, and wherever one click could either patch a region locally or commit to a whole style. The paired comparison between our policy and the supervised ablation settles this directly. If the gap is under a tenth of a click on both REFUGE2 and LIDC-IDRI, we will conclude that effort-aware optimization is unnecessary for current multi-rater benchmarks, and that result is itself worth reporting.
Why not just train the critic online with ordinary actor-critic? Nothing in the medical segmentation literature shows that co-training fails here. The evidence comes from control and language models, namely RED against GAIL and DVPO's report of critic drift during RLHF. The online-critic ablation, run with matched compute over five seeds, tests whether the frozen critic gives lower variance across seeds and fewer clicks. If the two perform the same, freezing is only a convenience.
Why not just use an ensemble of per-rater models and let the clinician pick one? The top QUBIQ entries were ensembles, with average Dice between 0.80 and 0.85 according to the QUBIQ summary. That same summary names their computational cost in 3D as a barrier, and asking a clinician to choose among K masks is itself an interaction. We will include a "choose the best ensemble member, then click" baseline, and count the choice as one interaction.
Why not just place clicks where raters disagree most? That heuristic decides where to ask for input, which is a different problem. In our setting the clinician decides where to click, and the model decides what mask to show. We keep the question narrow on purpose and do not claim to improve how clicks are placed.
Proposed Experiments
The main benchmarks are REFUGE2 (seven ophthalmologists, optic disc and cup), LIDC-IDRI (four radiologists, lung nodules) and the QUBIQ brain-growth task (seven experts). All three use leave-one-rater-out splits, so every reported number concerns a clinician the system never saw. The primary metric is mean clicks to 80, 85 and 90 percent Dice against the held-out rater's own mask. Secondary metrics are Dice after one to five clicks, first-mask Generalized Energy Distance, continuous Dice on QUBIQ, and the critic calibration gap. For QUBIQ brain growth, the variational model's reported continuous Dice of 0.5372, against 0.5083 for Bayes Dropout, anchors the single-pass comparison.
The first decisive check is the comparison with the supervised ablation described above. The second is robustness to the click simulator: we retrain nothing, but evaluate with a simulator that clicks on boundary errors weighted by distance, instead of at region centres. This tests whether the gains are an artefact of the policy learning to exploit one simulated clinician. The third is the critic calibration audit, which shows whether decoupling holds up once the policy leaves the offline data.
A small reader study, in which two clinicians refine 30 REFUGE2 cases with each system, would test whether gains measured on simulated clicks carry over to real ones. We treat it as confirmatory rather than primary.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- PseudoClick: Interactive Image Segmentation with Click Imitation
- MedUHIP: Towards Human-In-the-Loop Medical Segmentation
- Pretrain Value, Not Reward: Decoupled Value Policy Optimization
- Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
- Energy-Based Imitation Learning
- Optimal Transport for Offline Imitation Learning
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge