Can Experts Tell Real Rater Disagreement from Model Samples? Visual Lineup Tests for Multi-Rater Segmentation

When several radiologists outline the same lung nodule or prostate, their outlines differ, and a growing family of segmentation models tries to reproduce that disagreement by producing many plausible outlines instead of one. These models are ranked with numbers such as overlap averaged across probability thresholds or distances between sets of masks. No one has checked whether those numbers agree with what a clinician would see as a faithful copy of expert disagreement. We propose a direct test built from statistical graphics. Show a clinician a small panel of outline sets: one is the real set drawn by the human raters, and the rest were sampled from the model. Ask which one is real. If trained viewers cannot beat chance, the model's disagreement is visually indistinguishable from the experts'. If they can, the model is missing something a person notices.

The opportunity is that this test can serve as an external reference standard for the benchmark metrics themselves. Some widely used summaries depend only on how often each pixel is covered. They are provably blind to whether the outlines form coherent alternative shapes, two distinct hypotheses, or a smear. The same blindness applies to the pixel-uncertainty heatmaps shown to clinicians. The core claim is testable: displays that preserve whole-outline structure, such as animated cycling through individual outlines, let experts detect errors in modeled disagreement that per-pixel metrics and heatmaps cannot register. Human lineup verdicts can then tell the community which automated metric to trust on public multi-rater benchmarks.

Why this gap is real

Three lines of work motivate the question without answering it. The probabilistic segmentation models PHiSeg, a multiscale variational model of rater variation (PHiSeg), and OMBA-UNet, a shared encoder with one Bayesian decoder per known rater (Rater-Specific Bayesian Neural Networks), are judged entirely by offline distances and communicate uncertainty through static per-pixel maps. The QUBIQ challenge standardized multi-rater evaluation across six MRI and CT tasks with Q-Dice (QUBIQ). Q-Dice thresholds the predicted and the rater-averaged probability maps at nine levels and averages the resulting Dice scores. Because it only sees averaged maps, any two sample sets with the same per-pixel frequencies receive the same score. Zepf et al. (Navigating Uncertainty in Medical Image Segmentation) showed that the generalized energy distance (GED) conflates "is there a lesion" with "where is the boundary." They split it into a detection term and a boundary term, and showed that the split changes model rankings. Their paper is our closest methodological neighbour. It establishes that metric choice changes conclusions, but it offers no reference point for deciding which metric reflects clinically noticeable fidelity. That missing reference point is what we supply.

The visualization literature supplies the method. Hullman and Gelman (Theories of Graphical Inference) frame looking at a chart as an implicit check of observed data against data replicated from a model. A lineup makes that check explicit: the real data sit hidden among replicates. Hypothetical outcome plots, which animate individual draws from a distribution, cut viewers' error on probability-of-ordering judgments to about 10% mean absolute error, against 35 to 60% for error bars and violin plots (HOPs). That result was obtained on abstract one- and two-variable data, never on spatial contours. It also comes with a known cost of flicker and fatigue, which constrains our design toward short viewing times. Low-dimensional posterior projections (PPDE) show that averaging multimodal posteriors hides separate "islands" of probability. That work concerned image restoration, not rater disagreement. Finally, the inter-observer literature constrains the design. Diagnosis-first fusion (DiFF) and conformity filtering (Variability Matters) both collapse disagreement toward one "correct" answer, and the latter relies on an anchor rater assumed to be right. Our test deliberately takes no position on which rater is right. It asks only whether the model reproduces the spread.

Core Claim

Two hypotheses carry the proposal. First, when a model's samples match the real raters in per-pixel frequency but differ in joint shape structure, lineup detection by experts will exceed chance under a whole-outline display and stay at chance under a frequency heatmap. Q-Dice will likewise register no difference. Second, across real trained models on public benchmarks, rankings by expert lineup detection will agree more closely with some automated metrics than others. We expect them to agree with the decomposed GED terms and with a classifier-based two-sample test more than with Q-Dice. This agreement becomes a reproducible recommendation for which metric a benchmark should report.

The statistics are simple. In a lineup of $k$ panels with one real set, a viewer guessing at random picks the real set with probability $1/k$. With $n$ independent judgments of which $x$ are correct, the evidence that the model is distinguishable is the binomial tail

$$p = \sum_{j=x}^{n} \binom{n}{j} \left(\tfrac{1}{k}\right)^{j} \left(1-\tfrac{1}{k}\right)^{n-j}.$$

We report the detection rate $x/n$ minus $1/k$ as the model's "visual distinguishability," with bootstrap intervals over cases and readers.

Research Program

The first component is the stimulus generator. For each test case we take all real rater outlines: four on LIDC-IDRI, and six on QUBIQ prostate and the other QUBIQ tasks. We draw the same number of samples from each model, matched in count, and render every set identically after rater-order randomization. Using the same count matters because panel size alone could give the real set away.

The second component is a set of controlled corruptions with known ground truth. Each is applied to the real rater sets so that the true answer is certain. Mode collapse replaces every outline with the pixel-wise majority. Variance inflation dilates and erodes outlines beyond the observed spread, mimicking the overestimation Zepf et al. report for ensembles. The key marginal-preserving corruption computes the per-pixel rater frequency map and thresholds it at independent uniform random levels. This produces outlines whose per-pixel coverage matches the real set exactly, so Q-Dice and heatmaps are unchanged. The outlines are always nested, however, and cannot express shifted or competing shapes. Rater-style scrambling permutes which rater drew which outline across cases, so it tests whether per-rater structure of the kind OMBA-UNet models is visible.

The third component is the display factor: a static grid of small outlines, an overlaid frequency band equivalent to the heatmap, and an animated display cycling one outline set at a time at the few-hundred-millisecond cadence used for HOPs. Viewing is capped at a fixed window to limit fatigue. Readers are board-certified radiologists for CT and MRI tasks. As a secondary pool, we recruit trained non-clinicians to measure how much clinical expertise contributes.

The fourth component is the benchmark release. Case lists, rendered lineups, response logs and analysis scripts will be packaged so that any new multi-rater model can be scored against the same reference panels and the same automated metrics.

Baselines

The naive reference point is chance, $1/k$, together with the "do nothing" model: a single deterministic U-Net whose softmax output is thresholded at a sweep of levels to create pseudo-samples. This is exactly the nested construction above, so it doubles as a real-model instance of the marginal-preserving failure. Zepf et al. found that deterministic softmax entropy matched stochastic models on the low-ambiguity QUBIQ prostate task (no reported number found on Lacuna for that comparison). We therefore expect this baseline to be hard to catch on prostate and easy to catch on LIDC-IDRI. That dissociation is itself a check that the lineup measures ambiguity rather than rendering artifacts.

The strongest existing modeling baselines are PHiSeg, a deep ensemble, the Probabilistic U-Net, and OMBA-UNet. All are retrained from public code on identical splits with the same augmentation and a matched hyperparameter search budget, and all are sampled at equal counts. PHiSeg reports a GED of 0.2248 on LIDC-IDRI against 0.2393 for the Probabilistic U-Net, and 0.0864 against 0.1322 on prostate MRI (PHiSeg). Our retrained numbers must land near these values before any lineup is run, which guards against weak reimplementations. QUBIQ top teams report Q-Dice between 0.80 and 0.85 (QUBIQ). OMBA-UNet reports wins in 5 of 7 QUBIQ tasks, but no per-task number was found on Lacuna.

The competing evaluation baselines are Q-Dice, GED, the detection and boundary GED terms of Zepf et al., and a classifier two-sample test. That test trains a small network to distinguish real rater sets from model sample sets and reports held-out accuracy, with the same number of cases the human panels see. The proposal beats these metrics if one of two results holds. The first is that expert lineups detect at least one corruption that a given metric scores as indistinguishable from the real sets. The second is that expert rankings of the real models disagree with that metric's ranking beyond bootstrap uncertainty. The ablation isolating our key ingredient, whole-outline display, compares detection under the frequency-band display with detection under the grid and the animation, holding cases and readers fixed.

Why not just…?

Why not just use GED or its decomposed terms, which already capture joint structure? GED does compare sets of whole masks, so unlike Q-Dice it is not blind to the nested corruption. Zepf et al. show, though, that its components rank models differently, and nothing in the literature says which ranking matches what clinicians perceive. The lineup experiment settles this directly. If expert rankings track the GED terms, the outcome is a useful validation and a recommendation to report them, not a failure of the proposal. We expect to find that the boundary term aligns with expert judgments on LIDC-IDRI.

Why not just show static 50% and 90% confidence contours, which clinicians already find familiar? Nested confidence contours are functions of the per-pixel frequency map. By construction they cannot distinguish the real rater set from its marginal-preserving corruption, and PPDE's analysis of multimodal posteriors (PPDE) predicts exactly this averaging failure. The display ablation tests the claim empirically. If readers using frequency bands still detect the nested corruption, perhaps through cues we have not anticipated, the argument for animated displays weakens, and we will report that.

Why not just replace humans with the classifier two-sample test? This is the real threat. With only four to six raters per case and a few hundred test cases, such a classifier may be underpowered, or it may latch onto rendering quirks. Humans may instead ignore clinically irrelevant differences that the classifier flags. We treat it as a candidate proxy. If its accuracy correlates tightly with expert detection across corruptions and models, the recommended benchmark protocol becomes "run the classifier, and validate it periodically with lineups," which is cheaper and more reproducible.

Why not just collapse raters to a consensus mask and skip modeling disagreement? DiFF reports diagnosis AUC gains of up to 2.73% on glaucoma over majority-vote-style fusion (DiFF), evidence that rater structure carries signal. Conformity filtering works only under an anchor assumption its authors flag as unverified (Variability Matters). The mode-collapse corruption measures how visible consensus-only output is to experts on each benchmark.

Proposed Experiments

The first decisive check uses the corruption suite on LIDC-IDRI and QUBIQ prostate, kidney and brain tumor, crossed with the three displays. The prediction is sharp: the nested marginal-preserving corruption is detected above chance under grid and animation, sits at chance under frequency bands, and leaves Q-Dice unchanged. If it is undetected under every display, whole-outline structure is not perceptually salient at these rater counts, and the central claim fails.

The second check runs lineups on the five retrained models. We compute Kendall rank correlation between expert distinguishability and each automated metric, separately for the high-ambiguity dataset (LIDC-IDRI) and the low-ambiguity one (QUBIQ prostate). This follows Zepf et al.'s finding that conclusions differ between these regimes.

The third check uses rater-style scrambling on OMBA-UNet outputs. It asks whether per-rater decoders produce structure that experts can see. If they cannot, the 17-fold inference cost reported for that model buys nothing visible.

Power is planned for detecting a 15-point rise over the chance rate of a five-panel lineup at the per-condition level. Readers are counterbalanced across displays so that each case is seen once per reader.