When several experts outline the same organ or lesion, their outlines differ. The field now trains models that sample a spread of plausible outlines instead of one answer, and it scores those samples by how well they overlap the human ones. Overlap scores miss one property. Human disagreement has spatial structure: one radiologist tends to draw the whole prostate a few millimetres wider than another, so neighbouring boundary points shift together. A model can match the amount of disagreement at every voxel and still get this structure wrong. Dropout-based samplers do exactly that, producing boundary jitter that is independent from one voxel to the next (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation reports this failure qualitatively).
We propose to measure that structure by borrowing a statistic from brain imaging. Regional homogeneity (ReHo) asks, for each voxel of a resting-state functional MRI scan, whether the voxel and its 7, 19 or 27 nearest neighbours rise and fall together over time. It answers with Kendall's coefficient of concordance, a rank-agreement score between 0 and 1 (Alterations of Regional Homogeneity in Preschool Boys With Autism Spectrum Disorders). We replace the time axis with the rater axis. For each voxel in a zone of disagreement, we ask whether its neighbours agree on which rater drew the boundary furthest out and which drew it furthest in. The result is a map, computed without training, that separates coherent and systematic rater bias (high concordance) from local, unstructured tracing noise (low concordance). It gives a test of whether generated outline sets reproduce the structure of human disagreement, not only its amount.
Core Claim
Treat each rater as a "time point." Treat each voxel's signed distance to that rater's boundary (negative inside, positive outside) as the signal. Then the ReHo computation carries over without change. For a neighbourhood of $K$ voxels and $R$ raters, rank the raters within each voxel by signed distance. Let $S_j$ be the sum of rater $j$'s ranks across the $K$ voxels, and let $\bar S$ be the mean of the $S_j$. The disagreement-homogeneity score is
$$W = \frac{12 \sum_{j=1}^{R} (S_j - \bar S)^2}{K^2 (R^3 - R)},$$
which is Kendall's W with the $K$ voxels acting as judges and the $R$ raters as the items being ranked. We use the standard tie correction.
We make three claims. First, W maps restricted to the voxels where raters disagree reveal structure that per-voxel entropy, variance and Q-Dice cannot see. Second, generative segmentation models that score similarly on overlap differ sharply on this structure. Third, models with explicit per-rater latent codes should reproduce human W distributions much better than Monte Carlo dropout or plain ensembles. The new part is the use of local concordance across raters as a structural fidelity measure. No paper in the file computes any spatial-coherence statistic of rater disagreement.
What we deliberately do not do
The obvious combination would feed resting-state ReHo maps into a multi-rater brain tumour segmenter, so that the physiological signal could decide between disputed tumour margins. The file's closest work motivates this direction. The Resting-State fMRI Regional Homogeneity Metrics … Are Valid Indicators of Tumor-Related Neurovascular Uncoupling found lower ReHo on the tumour side of sensorimotor cortex in 12 patients (KCC-ReHo p = 0.02). The same paper also blocks the idea, because it concedes that low ReHo may reflect disrupted tumour tissue rather than suppressed blood-flow coupling. A ReHo prior therefore cannot tell core from margin. There is also no dataset in the file that pairs multi-rater outlines with resting-state fMRI. QUBIQ's brain tumour task and the structural inputs used by Multi-rater Prism contain only static MRI. A 12-patient cohort cannot support a segmentation claim. We therefore take ReHo's mathematical operator and its known sensitivity to neighbourhood size, and leave its physiology aside.
Research Program
The first step is to establish the statistic. Kendall's W is inflated when R is small, and QUBIQ tasks have only a handful of raters (7 for brain growth). We therefore calibrate every map against a permutation null. Rater identities are shuffled independently at each voxel, which destroys coherence but preserves the per-voxel spread. We report the excess of W over its null expectation. Following ReHo practice, we compute maps at 7-, 19- and 27-voxel neighbourhoods. Because ReHo's own open problem is the trade-off between noise in small clusters and blurred boundaries in large ones, we treat neighbourhood size as a variable to analyse rather than a fixed choice.
The second step is validation on simulated raters where the truth is known. We start from a consensus mask and generate raters from two parts. The first is a global per-rater dilation or erosion, which is coherent bias. The second is a spatially white or short-correlation boundary perturbation, which is unstructured noise. We then check that excess W tracks the share of variance that is coherent, and how fast it saturates.
The third step describes real human data. We compute W maps on the QUBIQ tasks (prostate, brain growth, kidney, pancreas, pancreatic lesion and brain tumour; QUBIQ). We extend the measure to ordinal labels on Gleason 2019, where Gleason grades provide the ranking directly. We then ask whether coherence differs systematically by anatomy, for example between crisp organ edges and infiltrative lesions.
The fourth step uses W as a test of generative models. For each model, we draw exactly R samples per image, matching the human rater count so that W is comparable. We then compute the 1-Wasserstein distance between the model's and the humans' distributions of excess W over disagreement voxels. A secondary, exploratory step adds a differentiable surrogate of this distance as an auxiliary training loss. We pursue it only if the measurement results justify it.
Baselines
The naive reference is a single deterministic U-Net whose sigmoid output is thresholded at R evenly spaced levels to produce "samples." These samples are perfectly nested, so their W is degenerate. They give the floor for coherence that comes from smooth probability maps alone.
The strongest competitors on overlap are deep ensembles of deterministic U-Nets. These won QUBIQ with average Dice between 0.80 and 0.85 (QUBIQ). We include them because they are the method a sceptic would deploy. Ensemble members trained on individual raters' masks may inherit per-rater bias and so look coherent for free.
The strongest structured competitor on brain growth is the variational model of Variational Inference for Quantifying Inter-observer Variability. It reports a Continuous Dice of 0.5372 on QUBIQ brain growth, against 0.5083 for Bayes dropout and 0.5042 for the ESULR baseline. We also run Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability), which reports Cohen's κ = 0.758 on Gleason 2019 and 0.711 on the 25-rater breast set. Its per-rater Gaussian latents explicitly separate bias (the mean) from inconsistency (the variance), so it is the model our measure should favour if the measure is right. Monte Carlo dropout is the model it should penalise.
All models share one backbone (a U-Net, with TransUNet, Beckschen/TransUNet, as a check). They use the same splits, the same training budget and equal hyperparameter search trials, and the same sample count R. For the measurement claim, "beating" means the following. Excess W must separate models whose Continuous Dice or Q-Dice differ by less than 0.01. It must agree with a blinded reader study better than any overlap metric does. The ablation that isolates the key ingredient keeps everything fixed but replaces rank concordance with per-voxel entropy or variance computed over the same neighbourhood.
Why not just…?
Why not just use per-voxel entropy or variance maps? These measure how much raters disagree at a voxel. They are unchanged if rater identities are shuffled from voxel to voxel. That shuffle is exactly our null, so by construction they cannot detect coherent bias. The ablation above tests whether this blindness matters in practice: if dropout and the variational model have matching entropy maps but different excess W, it does.
Why not just use spatial autocorrelation, such as Moran's I, on each rater's deviation from the mean? This is the real threat. Per-rater residual autocorrelation targets the same property, and on QUBIQ it may rank models identically. Rank concordance has two advantages that we expect but have not shown. It does not depend on raters' differing scales, and it needs no reference mean, which is itself unstable at disputed edges. We will compute both on the simulations and on all real tasks. If Moran's I matches W on simulation recovery and on reader agreement, we will report that. The contribution would then reduce to the evaluation protocol, meaning matched-R sampling, the permutation null and the distance between W distributions, rather than the particular statistic.
Why not just rely on Pionono-style latents, which already split bias from noise? Pionono's split is per rater and global, it requires training, and it is defined only for the model's own outputs. A training-free map can audit any sample set, including human data and ensembles, and can localise where coherence breaks down. Agreement between Pionono's learned bias magnitudes and our measured coherence on the 25-rater breast data would cross-validate both.
Why not just keep Q-Dice, since QUBIQ already standardised it? Q-Dice thresholds the mean probability map, so it discards sample identity before scoring. Any two sample sets with the same mean map receive the same score. The decisive experiment below tests whether that loss matters to clinicians.
Proposed Experiments
The decisive checks come in three parts. First, on simulated raters, excess W must increase monotonically with the true share of coherent variance at every neighbourhood size, and its permutation-null distribution must be centred near zero for R between 3 and 7. Second, across the QUBIQ tasks and Gleason 2019, MC dropout must show excess W markedly below the human value. Pionono and the variational model must come closer to the human value, while their Continuous Dice differences stay small. If dropout matches human coherence, the motivating premise is wrong.
Third, two clinicians who did not annotate QUBIQ will see, blinded, a human rater set and a model sample set for the same image, and will choose which looks like independent human tracings. We then compare how well excess-W distance, Moran's I, entropy distance and Q-Dice predict their choices. The proposal succeeds if excess W, or its Moran's I counterpart under the outcome described above, predicts reader choices clearly better than Q-Dice and entropy. If no structural statistic beats overlap, we will conclude that the spatial structure of disagreement, though measurable, does not affect how plausible outlines look to clinicians.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Alterations of Regional Homogeneity in Preschool Boys With Autism Spectrum Disorders
- The Resting-State Functional Magnetic Resonance Imaging Regional Homogeneity Metrics - Kendall's Coefficient of Concordance-Regional Homogeneity and Coherence-Regional Homogeneity - Are Valid Indicators of Tumor-Related Neurovascular Uncoupling
- Multi-rater Prism: Learning self-calibrated medical image segmentation from multiple raters
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures