Spending Expert Attention Where Segmentation Disagreement Changes the Diagnosis

When several doctors outline the same tumor, optic cup, or lesion, they draw different boundaries. An AI system trained on one doctor's outlines learns that doctor's habits, and it does not average them away. It exaggerates them, and it does so with confidence. In deployment, a hospital has a small budget of senior-expert time for checking the AI's work. The usual way to spend that budget is to send the expert the cases where the model looks most uncertain. This proposal argues that this rule fails in exactly the setting that matters. A model that has absorbed a junior annotator's systematic bias is confidently wrong, so its uncertainty points the expert away from its worst mistakes.

We propose treating expert review as an exploration–exploitation problem, the classic choice between acting on what you already believe and gathering information that might change your beliefs. The system keeps a small posterior (a probability distribution that it updates as evidence arrives) over how the expert's boundaries differ from the ones the model learned. It decodes plausible expert outlines from that posterior and asks where they would change the downstream diagnosis. It then sends the expert the cases whose answers either fix a diagnosis now or teach it the most about the bias that affects every future case. In two sentences: disagreement between observers matters only where it changes the diagnosis, and a model trained on biased labels cannot see its own bias from its uncertainty. So expert queries have to be allocated partly to explore that bias, not only to resolve the cases that already look ambiguous.

Core Claim

Three findings in the literature combine into a gap. First, The Impact of an Inter-rater Bias on Neural Network Training shows that 3D U-Nets trained on one rater's MS-lesion or hemorrhage masks become more distinguishable from the other rater than the humans were. A classifier separates the networks' outputs at 0.90–0.93 versus 0.87–0.917 for the human masks. The same paper shows that adding a few senior annotations to the training set reduces the bias. This paper motivates the proposal: the error is systematic and has low entropy.

Second, Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle (DiFF) weights multi-rater masks by how well they support a downstream diagnosis network, and improves AUC over earlier multi-rater methods on glaucoma, thyroid, and melanoma. It supplies our objective, which is diagnostic utility rather than overlap. It is static, however: it reweights labels that already exist and never decides what to ask for.

Third, Task-optimal exploration in linear dynamical systems (TOPLE) proves two things. Exploration aimed at general model uncertainty wastes budget on directions the task does not care about. Low-regret greedy learners fall into a "regret trap": they stop exploring directions needed to certify the optimal decision. That regret trap is precisely what an uncertainty-deferral rule does to a biased segmenter.

The claim is as follows. Under a fixed budget of expert queries, a policy that runs Thompson sampling over a low-dimensional expert-bias posterior and scores cases by expected change in diagnosis should reach higher expert-agreed diagnostic accuracy than three alternatives: entropy-based deferral, greedy diagnosis-flip deferral, and random explore-then-commit. (Thompson sampling means drawing one plausible hypothesis from the posterior and acting as if it were true.) We expect the margin to grow with the size of the training rater's systematic bias. When the bias is zero, we expect the method to collapse to greedy deferral, and we will report whether it does.

Research Program

The base model is a rater-conditioned probabilistic segmenter in the style of Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono). Pionono gives each rater a Gaussian in an 8-dimensional latent space whose mean encodes that rater's systematic bias, and decodes masks through late injection. We reuse this architecture unchanged. Its purpose here is to give us a compact place where "the expert's style" lives. A downstream diagnosis function $d$ maps a mask, and optionally the image, to a clinical decision. Examples are the vertical cup-to-disc ratio thresholded for glaucoma on REFUGE-2, a lesion-load or volume threshold on MS and hemorrhage data, and the melanoma or thyroid-malignancy classifiers used by DiFF.

At deployment we have a model trained on "junior" raters and a stream of cases. An expert oracle can be queried $k$ times per batch, and each query returns the expert's mask for one case. Following the Collect & Infer split in Collect & Infer - a fresh look at data-efficient Reinforcement Learning, collection is separate from inference. Queried cases receive the expert's answer, and all other cases receive the model's diagnosis under the current posterior. The posterior is over the offset $\delta$ between the expert's latent mean and the training latent, and it is updated from each returned mask through the decoder likelihood. For each candidate case $x$, the policy draws $\delta \sim p(\delta \mid \mathcal{D}_t)$, where $\mathcal{D}_t$ is the set of expert answers so far. It then computes

$$ g(x) = \Pr\big[d(s_\delta(x)) \neq \hat d_t(x)\big], $$

where $s_\delta(x)$ is the mask decoded with the sampled expert offset and $\hat d_t(x)$ is the diagnosis the system would otherwise report. The policy queries the top-$k$ cases by $g$.

The sampling step produces the exploration. While the posterior over $\delta$ is wide, some draws imply that the expert's outlines are systematically larger or smaller than the model's. Those draws flag cases the model is confident about, because a consistent shift moves many near-threshold cases across the decision boundary. The weighting by $d$ provides the task focus. Boundary disagreements that move no diagnosis contribute nothing, in line with TOPLE's argument against task-agnostic exploration. We will also test an explicit variant with a term for information about $\delta$, weighted by how many pool cases lie near the decision threshold. We keep it only if it beats plain Thompson sampling.

Baselines

The do-nothing reference is the junior-trained model reporting its own diagnoses with no expert queries, alongside a random-query policy that spends the same $k$ queries uniformly. Any method has to beat random at equal budget to justify itself.

The strongest existing competitor for choosing what to send to review is entropy-based deferral as implemented in The Latent Doctor Model for Modeling Inter-Observer Variability. That paper flags cases whose predictive entropy is high for human review and shows that accuracy improves on the cases that remain. Its entropy also tracks real panel disagreement, with Spearman correlation up to 0.728 against 20 pathologists on ISUP grading. We run it on the same Pionono backbone, the same budget, and the same diagnosis function. No reported result was found on Lacuna for its diagnostic accuracy at a fixed expert-query budget, so we measure it ourselves. Beating it means higher expert-agreed diagnostic accuracy at every budget from 1% to 20% of cases, with non-overlapping bootstrap intervals over seeds and rater-assignment splits.

The strongest static competitor for label quality is DiFF. It reports AUC improvements of up to 2.73% (glaucoma, REFUGE-2), 2.51% (thyroid, TNMIX) and 2.04% (melanoma, ISIC) over earlier multi-rater methods. It reports these as relative gains, and no absolute AUC appears in the file. As an external anchor for glaucoma, the REFUGE challenge reports a top classification AUC of 0.9885 (REFUGE challenge). We train DiFF on the same junior-rater data and report it at zero queries, plus DiFF followed by our query policy. This tests whether query allocation adds value on top of a better static label model. For rater-style fidelity itself, Pionono's reported unweighted kappa of 0.758 on Gleason 2019 is the bar our backbone must reproduce before any query experiments count.

Two ablations isolate the key ingredient. The first is greedy diagnosis-flip deferral: the same score $g$, but computed with the posterior mean of $\delta$ instead of a sample, or equivalently with sampling switched off. This removes exploration and keeps task focus. The second is Thompson sampling with a task-agnostic score, meaning expected pixel disagreement instead of diagnosis flips. This removes task focus and keeps exploration. We also include a relaxed-UCB selector adapted from Training Multi-organ Segmentation Networks with Sample Selection by Relaxed Upper Confident Bound, which samples randomly within a high-score band to avoid getting stuck on a few extreme cases. All baselines share backbone, tuning budget, query budget, and seeds.

Why not just…?

Why not just defer the most uncertain cases, as LDM does? Uncertainty deferral does work when disagreement shows up as ambiguity. The bias-amplification result shows the opposite failure: the trained network's errors are more consistent than the human rater's, so they will not register as high entropy. We test this directly by measuring what fraction of expert-diagnosis errors fall among the model's lowest-entropy cases. If that fraction is small on real data, this proposal's premise is weak and entropy deferral wins.

Why not just use greedy diagnosis-flip scoring without sampling? This is the most serious threat, because it already captures the "only care about diagnosis-changing disagreement" half of the idea. TOPLE's regret-trap argument predicts that it stops querying confident cases and never learns a global shift. That is a prediction, though, not evidence in this domain. The greedy ablation settles the question. If greedy matches Thompson sampling across all bias magnitudes, we drop the exploration claim and report a task-focused deferral method instead.

Why not just label a random handful of cases with the expert up front and fine-tune? This is mixed-source training from the bias paper, and it is an explore-then-commit strategy of the kind TOPLE argues is sometimes necessary. It is a real competitor. We run it at matched budget, with random selection of the up-front set followed by fine-tuning of the latent offset. If it matches our method, the practical conclusion is that the choice of cases matters little, which is itself useful to know. We expect it to lose when bias affects only near-threshold cases, which random selection rarely hits.

Why not just use a better consensus label, such as STAPLE or majority vote? Consensus labels cannot help when the expert differs systematically from the whole junior panel, and DiFF already shows that majority vote underperforms diagnosis-aware labels. We include a majority-vote-trained model as a zero-query reference.

Proposed Experiments

We simulate deployment on multi-rater data by assigning one rater, or a subset of raters, to training and holding out a designated expert as the oracle. The ISBI 2015 MS data is the most natural case, since its two raters have 4 and 10 years of experience; the diagnosis there is lesion-load category. On LIDC-IDRI, one of the 4 radiologists is the oracle and the diagnosis is a nodule-volume threshold. On REFUGE-2 the diagnosis is glaucoma from the cup-to-disc ratio. We use Gleason 2019, with 6 pathologists, for grade-level decisions, and ISIC or TNMIX where the multi-rater masks are available. We rotate the oracle role across raters and report the mean with its spread.

The primary metric is diagnostic accuracy against the oracle expert as a function of query budget, together with the number of queries needed to reach 95% of full-expert accuracy. Secondary metrics are the fraction of misdiagnoses occurring on low-entropy cases, generalized energy distance to the oracle's masks, and Pionono kappa as a check on the backbone.

The decisive check is a controlled bias sweep. On REFUGE-2 and LIDC-IDRI, we dilate or erode the training rater's masks by a fixed 0–3 pixels to create known systematic bias. We then plot the accuracy gap between Thompson sampling and greedy deferral against bias size. The claim predicts a gap near zero at zero bias that grows with bias. A flat gap falsifies it. A second check tests whether the posterior over $\delta$ recovers the injected shift. A third tests whether Thompson sampling's advantage comes from querying confident cases, which we measure by the entropy distribution of the cases each policy chose.

We state the scope plainly: these are retrospective simulations with rater identities as a proxy for expertise. A reader study along the lines of Luo et al.'s BMSS trial, in which residents gained more from assistance than attendings, would be the follow-up that tests whether the allocation also helps real experts.