When several radiologists or pathologists outline the same tumor or organ, their contours rarely agree. The disagreement is not random. Some experts consistently draw wide and some draw tight, and all of them diverge most where the structure fades into its surroundings with little change in brightness. Models that learn each expert's style, such as Pionono and the Latent Doctor Model, currently need every expert to trace full outlines on every training image, which multiplies the already severe cost of expert labeling by the number of experts. Cheap labels do not help much either. Scribbles, which are quick strokes drawn through the middle of a structure, sit far from the edge, which is exactly where experts disagree. They therefore say almost nothing about a rater's style.
We propose to measure rater style where it actually shows up. Each expert places a handful of clicks marking where they believe the edge lies, at boundary locations the system chooses because the local contrast there is low. The model describes each rater with a few interpretable numbers: a constant inward or outward shift (their aggressiveness), a sensitivity term that scales that shift by how faint the edge is, and a noise level for their own inconsistency. Intensity standardization, meaning the mapping of scanner-dependent brightness values onto a common scale, is what makes "how faint the edge is" comparable across scanners. The claim is that this separates human style from scanner-induced contrast, and that roughly a dozen well-placed clicks per rater can recover per-rater predictions close to those of a model trained on full multi-rater tracings.
Core Claim
Inter-observer disagreement in segmentation is concentrated in low-contrast boundary segments. A rater's systematic behavior there is well approximated by a low-dimensional, contrast-dependent displacement of a shared boundary. If that holds, three consequences follow. Rater models can be fit from sparse boundary clicks instead of dense masks. The best places to ask a rater are chosen by contrast, not at random. And once intensities are standardized, a rater's parameters estimated on one scanner should transfer to another, because scanner effects enter only through the measured contrast and not through the rater's own parameters.
Formally, let a shared network predict a consensus signed distance field $\phi(x)$, which is positive inside the structure and negative outside, for each location $x$. Rater $r$'s predicted mask is the set where
$$\phi(x) + a_r + b_r\, h\big(c(x)\big) + \epsilon_r(x) > 0,$$
where $c(x)$ is the intensity step across the boundary normal nearest $x$, measured on standardized intensities. Here $h$ is a fixed decreasing function, so fainter edges allow larger shifts. The parameter $a_r$ is the rater's constant aggressiveness, $b_r$ is their sensitivity to faint edges, and $\epsilon_r$ is zero-mean noise with rater-specific scale $\sigma_r$ that captures intra-observer inconsistency. Each click by rater $r$ along a normal line directly observes the shift at that point, so fitting $(a_r, b_r, \sigma_r)$ is a small regression problem. Clicks at low-contrast points are the most informative about $b_r$.
Relation to Prior Work
Two latent-variable observer models motivate this work and also constrain it. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater an 8-dimensional Gaussian whose mean captures systematic bias and whose variance captures inconsistency. It reaches unweighted Cohen's κ of 0.758 on Gleason 2019 and 0.711 on a 25-rater breast cancer set. The Latent Doctor Model for Modeling Inter-Observer Variability encodes a rater from a support set of their labeled cases with a DeepSet, a network that summarizes an unordered collection, and conditions the prior on the image to flag ambiguous cases. Both show that disagreement is structured rather than noise. We borrow that premise and Pionono's mean-versus-variance split. Both models, however, consume dense labels from every rater, and both use an opaque latent that cannot say whether a rater shifts because of habit or because the image is faint. Our model gives up expressive latents in exchange for an explicit contrast axis and a label format that costs a few seconds per rater per image.
The scribble literature supplies the cheap-label machinery, and it shows why scribbles are the wrong signal here. Learning to Segment Medical Images with Scribble-Supervision Alone grows scribbles with random walks and a conditional random field (CRF, a smoothing step that groups pixels of similar brightness), and filters errors with Monte Carlo dropout uncertainty. It reaches 88.0% Dice on ACDC and 78.1% on NCI-ISBI prostate. ZScribbleSeg corrects the tendency of scribble-trained models to under-segment using mixture-ratio priors and shape losses (0.862 Dice on ACDC). Learning to Segment From Scribbles Using Multi-Scale Adversarial Attention Gates learns shape priors from unpaired masks (84.3% Dice on ACDC). All three treat the scribble source as a single objective annotator. Their growth steps also depend on intensity differences, which leak at faint edges. We use them only to produce the shared field $\phi$ and never to infer rater style.
Two further threads supply the representation and the axis. Bounding Maps for Universal Lesion Detection shows that soft, graded targets beat hard edges for low-contrast lesions (+3.85% sensitivity at 4 false positives per image on DeepLesion). This supports representing the boundary as a continuous signed field rather than a hard contour loss of the kind used in A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S). It also addresses the tension noted in the literature between sharp contour regularization and multi-rater uncertainty: each rater's contour is sharp, and the spread comes from the offsets. The Influence of Intensity Standardization on Medical Image Registration shows that non-standard intensity scales distort gradient-based costs at tissue interfaces. Myronenko et al. show that organ-adaptive normalization outperforms static global scaling. We take from both that $c(x)$ is meaningless unless intensities are standardized, and preferably standardized locally, since large pathologies skew global histograms.
Research Program
The first stage is a diagnostic that could falsify the premise cheaply. On the dense multi-rater datasets we decompose each rater's deviation from consensus into two parts. One is boundary displacement, meaning signed distance along normals. The other is region relabeling, meaning disconnected areas assigned a different class. We then regress displacement on standardized local contrast. The premise requires two things: displacement must dominate, and it must grow as contrast falls. On histopathology grading, where much disagreement is about grade 3 versus grade 4 rather than edge position, relabeling may dominate. If so, we report it and treat the radiology arm as primary. In that case the model gains a per-rater class-confusion term for relabeled regions, but we do not claim clicks can recover it.
The second stage builds the model. The shared field $\phi$ is trained from consensus masks where they exist, or from scribbles using the Can et al. pipeline in the label-scarce setting. Local contrast $c(x)$ is computed after standardization: bias correction plus organ-adaptive histogram landmarks for MRI, and stain normalization for histology. Rater parameters are fit by maximum likelihood from clicks, with a population prior over $(a, b, \sigma)$ so that raters with few clicks shrink toward the average rater.
The third stage is query selection. For each rater, the system picks the boundary normals that most reduce posterior variance of $(a_r, b_r)$ given current estimates. In practice this favors a spread over the low-contrast range. We compare this against random boundary points.
Proposed Experiments
The main evaluation uses Gleason 2019 (6 pathologists) and the 25-rater breast cancer segmentation set, the two multi-rater dense segmentation benchmarks available. Because full tracings exist, clicks can be simulated exactly: a click is where rater $r$'s mask crosses a queried normal, plus small placement jitter. For each rater we hold out images and score predicted per-rater masks with Dice and with unweighted κ, which matches Pionono's reporting. We also score calibration of disagreement: the Spearman correlation between predicted pixel-wise spread across raters and empirical spread. This mirrors the entropy correlation that the Latent Doctor Model reports for classification on ISUP (ρ = 0.728). That number is for image-level grading, so it is a reference only and not directly comparable. The budget axis is the number of clicks per rater, set to 3, 6, 12, 24, and 48. Dense-mask budgets are converted to clicks through the boundary vertex count of a traced outline. This is a crude proxy, and it is replaced by wall-clock time in the radiology collection.
The radiology arm addresses the actual low-contrast setting. We will recruit a small panel of readers to click boundaries on NCI-ISBI 2013 prostate MRI, where the central gland and peripheral zone interface is faint, and on ACDC myocardium. Each reader also fully traces a small subset, which serves as ground truth for that reader and records real annotation time. Scanner transfer is tested by fitting rater parameters on one acquisition group and predicting on another, where acquisition metadata allows such a split. Where it does not, we test transfer under controlled monotone nonlinear intensity remappings of the test images. With standardization, rater parameters should stay stable under such remappings. Without it, they should drift.
Baselines
The naive reference predicts the consensus mask for every rater. It is a U-Net trained on majority-vote or STAPLE labels. Any rater model must beat it on per-rater κ; otherwise individual style is not being captured at all. A slightly stronger naive reference adds Monte Carlo dropout to the consensus model, as in Can et al. This produces spread without rater identity and tests whether generic uncertainty already tracks human disagreement.
The strongest existing method is Pionono trained on full dense masks from every rater. It reports κ = 0.758 on Gleason 2019 and κ = 0.711 on the breast cancer set in Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation. It is an upper reference trained with far more labels. We do not expect to beat it outright. Success means coming within 0.02 κ of it at 12 clicks per rater, and matching it in the disagreement-spread correlation. We will rerun it with the same backbone, splits, preprocessing, and tuning budget rather than relying only on the published figure.
The fair competitor is Pionono, or a Latent Doctor Model style support-set encoder, given a small number of dense masks per rater at a click-equivalent budget. The shared backbone is trained on the same consensus data. Each rater's latent is fit on $k$ fully traced images, with $k$ matched to our click budget. Beating it means higher per-rater κ at equal budget across the budget curve, most importantly in the low-budget regime. No reported result found on Lacuna for either model in a few-shot per-rater setting.
Three ablations isolate our ingredients. Setting $b_r = 0$ leaves only a constant per-rater offset and tests whether the contrast term matters. Removing standardization, so that $c(x)$ is computed on raw intensities, tests whether standardization is what makes rater parameters transfer. Replacing contrast-guided queries with random boundary points tests query selection.
Why not just…?
Why not just fit Pionono from a few dense tracings per rater? This is the real threat. Pionono's latent is only 8-dimensional, and the Latent Doctor Model already builds rater embeddings from small support sets, so a few full tracings might pin down a rater as well as our clicks do. The budget-matched comparison above settles it directly. Our bet rests on efficiency: a full tracing spends most of its vertices on high-contrast stretches where raters agree, while our clicks concentrate on faint stretches. If few-shot Pionono matches us at equal real annotation time in the radiology collection, the contribution shrinks to interpretability and scanner transfer, and we will say so.
Why not just collect scribbles from each rater and run a scribble method per rater? Scribbles lie in the interior, while disagreement lives at the edge, so a rater's scribbles carry little information about how wide they draw. Methods such as ZScribbleSeg also add shape and size priors that pull every rater toward the same shape. We will test this by simulating per-rater scribbles from the dense masks. We predict that per-rater κ stays near the consensus baseline.
Why not just standardize intensities offline and train existing observer models unchanged? Standardization alone makes scanners comparable, but it does not tell the model that disagreement depends on contrast. An opaque latent can still absorb site effects whenever raters and sites are correlated. The no-contrast-term ablation, run on standardized data, measures what the explicit contrast axis adds. The scanner-transfer test measures whether rater parameters stay fixed when only contrast changes.
Why not just train on consensus labels and report model uncertainty? Consensus training discards who disagrees and in which direction. The Latent Doctor Model already finds that majority-vote training underrepresents the spread of the reference label distribution. Generic dropout uncertainty also tracks model ignorance rather than human disagreement. The dropout-augmented consensus baseline and the spread-correlation metric test this directly.
Decisive Checks
The project hinges on three outcomes. First, the diagnostic must show that boundary displacement grows as standardized contrast falls, at least in the prostate and cardiac MRI arm. Second, contrast-guided clicks must beat budget-matched few-shot Pionono on per-rater κ. Third, rater parameters must stay stable under intensity remapping when standardization is applied and drift without it. Failing the first falsifies the premise. Failing the second leaves only an interpretable reparameterization of existing models. Failing the third means the separation of rater style from scanner contrast is not real.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Learning to Segment Medical Images with Scribble-Supervision Alone
- ZScribbleSeg: Zen and the Art of Scribble Supervised Medical Image Segmentation
- Learning to Segment From Scribbles Using Multi-Scale Adversarial Attention Gates
- Bounding Maps for Universal Lesion Detection
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- The role of intensity standardization in medical image registration
- Aorta Segmentation from 3D CT in MICCAI SEG.A. 2023 Challenge