Where Experts Disagree: Lorenz Curves for Inter-Observer Uncertainty in Segmentation

When several radiologists or pathologists outline the same tumor or organ, their tracings differ. This is inter-observer variability, and the field now trains models that predict a spread of plausible outlines instead of a single one. These models are scored with averages over a test set: mean staged Dice, mean calibration error, mean distance between the predicted and annotated distributions. An average does not show where the disagreement sits. Plausibly, most scans are ones experts trace almost identically, and a small minority of genuinely ambiguous cases holds nearly all of the disagreement. If that is true, a model can post a strong mean score by being sharp on easy cases while spreading its uncertainty thinly and in the wrong places. Clinically, the ambiguous minority is the part that matters.

This note borrows the economist's tool for "who holds the wealth": the Lorenz curve and the Gini coefficient, a 0–to–1 score of how concentrated a quantity is across a population. The population becomes cases, and the wealth becomes human disagreement or model uncertainty. In two sentences, the proposal is this. First, measure how concentrated human disagreement is across cases. Then check whether a model's predicted uncertainty is concentrated on the same cases to the same degree, and whether that alignment reorders methods that look equivalent under mean metrics.

Core Claim

We claim three things. Case-level inter-observer disagreement on standard multi-rater benchmarks is strongly unequal. Current multi-rater segmentation models misallocate their uncertainty relative to that inequality, either too flat or too peaked. And a concentration-curve evaluation detects this misallocation and ranks methods differently than mean Q-Dice or mean generalized energy distance do.

The practical reading is triage. The curve's height at fraction $k$ answers the question: if a clinic sends the $k\%$ of cases the model is least sure about to a second expert, what share of all real expert disagreement does it catch?

The research file reports no prior work that computes a Gini coefficient over inter-observer segmentation variability. The closest precedent is in a different domain. The Devil is in the Tails uses Gini through its LTAnalyzer tool to show that software-engineering label distributions are highly skewed ($G$ between 0.77 and 0.93), and that aggregate accuracy hides tail collapse. We carry that diagnostic logic over to annotator disagreement. We add the piece that work lacks: comparing two distributions over the same population, namely where humans disagree versus where the model is uncertain.

Research Program

For each case $i$ among $n$ test cases with several expert masks, we compute a human disagreement score $d_i$. We use two versions.

The first is a normalized one: mean pairwise $1-\text{Dice}$ among raters. It is comparable across structure sizes.

The second is an absolute one: the voxel count in the union minus the intersection of rater masks. It reflects the clinical volume at stake.

We report both, because the Gini coefficient is invariant to rescaling every value by the same factor but not to the choice of what is measured. Tiny lung nodules and large organs will rank differently under the two versions.

For each model we compute an uncertainty score $u_i$ of the same type. For sampling models such as the Mixture of Stochastic Experts (MoSE), Pionono, Probabilistic U-Net and PHiSeg, $u_i$ is mean pairwise $1-\text{Dice}$ among drawn samples. For models that output only a soft probability map, such as ensembles or networks trained with Dice Semimetric Losses, we use a map-level statistic applied identically to the averaged human map. This statistic is the disagreement mass $\sum_p q_p(1-q_p)$ normalized by expected foreground, where $q_p$ is the predicted foreground probability at pixel $p$. Applying the same statistic to both humans and models removes any advantage from one model family's output format.

The inequality of a nonnegative quantity $x$ over cases is

$$G(x) = \frac{\sum_{i}\sum_{j} |x_i - x_j|}{2 n^2 \bar{x}},$$

where $\bar{x}$ is the mean of $x$. We apply the $n/(n-1)$ small-sample correction and use bootstrap intervals over cases, since several QUBIQ tasks have few test cases.

Alignment is measured with the concentration curve $L_{u|d}(k)$. This is the share of total model uncertainty held by the fraction $k$ of cases ranked lowest by human disagreement. Its index is

$$C_{u|d} = 1 - 2\int_0^1 L_{u|d}(k)\,dk.$$

$C_{u|d}$ equals $G(u)$ exactly when the model ranks cases the same way humans do. It falls toward zero or below when the model's uncertainty sits on the wrong cases.

Each model therefore gets three numbers. The first is the concentration gap $G(u)-G(d)$, which says whether the model is too flat or too peaked. The second is the alignment ratio $C_{u|d}/G(u)$, which says whether the model's uncertainty is on the right cases. The third is the full curve. We report the full curve because, as the file notes, a single Gini value cannot distinguish distributions with different tails.

The program has three parts. The first is descriptive: Lorenz curves of $d$ across QUBIQ tasks, LIDC-IDRI, Gleason 2019 and the 25-rater breast cancer set, reporting how much of the total disagreement the top 10% of cases hold.

The second is evaluative: re-scoring the method families listed above under one shared protocol.

The third is a probe of one cheap intervention. We oversample cases in proportion to human disagreement during training and test whether this closes the concentration gap. The Devil is in the Tails found that naive re-weighting on high-Gini data often hurt, so we treat this as a hypothesis the experiment may reject, not a fix.

A secondary analysis tests a tension the file records. QUBIQ reports that simple averaged ensembles beat probabilistic architectures. MoSE and Pionono argue that averaging erases real alternative hypotheses. If the ensemble advantage on mean scores comes from easy cases while the probabilistic models are better aligned on the high-disagreement tail, the two positions are reconciled. If not, the ensemble win is broader than the multi-hypothesis literature admits.

Baselines

The naive reference is the current reporting practice: mean Q-Dice on QUBIQ, mean generalized energy distance on LIDC-IDRI, and expected calibration error. The question is whether our curve changes any conclusion these means support. To count as beating it, the curve must reorder at least one pair of methods whose mean-score intervals overlap, and that reordering must agree with the triage outcome described under Proposed Experiments.

The strongest competing evaluation tool is the per-case Spearman rank correlation between $u_i$ and $d_i$. We pair it with stratified reporting, meaning metrics on the top decile of cases by human disagreement. Together these are the obvious, cheap way to ask "is uncertainty on the right cases." Both are computed from exactly the same $u$ and $d$ vectors, bootstrap resamples and test splits as our indices, so the comparison is free of extra-information advantage. The concentration measures beat this baseline only if they separate methods that Spearman ties or misorders, and only if that separation predicts triage yield.

The concentration measures can in principle add something because Spearman ignores magnitude. A model that ranks cases perfectly but assigns nearly equal uncertainty to all of them gets a high Spearman score, yet it is useless for deciding how much review effort to spend.

The models being evaluated are the strongest existing methods in the file, each trained with the authors' published configuration and equal tuning budgets per task. The deterministic ensemble stands in for QUBIQ's winning style, since top QUBIQ teams reached average Dice of 0.80–0.85. The DML-trained network reduced calibration error on Prostate T1 from 0.51 under majority voting to 0.20. MoSE outperformed PHiSeg and Probabilistic U-Net on LIDC-IDRI generalized energy distance, but no reported numeric GED result was found on Lacuna for MoSE. Pionono reached Cohen's kappa of 0.758 on Gleason 2019 and 0.711 on the breast cancer set. The variational inference model reports continuous Dice of 0.5372 on QUBIQ brain growth, against 0.5042 for its ESULR baseline.

Two ablations isolate the key ingredient. The first swaps Gini for the coefficient of variation in the concentration gap, to test whether the bounded, cumulative-share form matters. The second evaluates models with the normalized versus the absolute disagreement definition, to show how much conclusions depend on structure size.

Why not just…?

Why not just report Spearman correlation between model uncertainty and rater disagreement? This is the real threat. If the concentration index and Spearman produce the same method ranking on every dataset, the Lorenz machinery adds presentation and nothing more, and we will say so. The experiment that settles it is rank agreement across methods (Kendall's tau between the two rankings), combined with the triage test. The triage test is the arbiter: whichever measure better predicts the share of disagreement caught at fixed review budgets is the more useful one. Our specific prediction is that sampling models with near-constant sample spread will show high Spearman but low $G(u)$, and will perform poorly at triage.

Why not just report metrics on the hardest decile? Worst-decile Dice picks one arbitrary cut and measures segmentation accuracy, not whether uncertainty is allocated correctly. A model can be accurate on hard cases yet fail to signal that they are hard. We include stratified scores as a baseline. The curve is the same information integrated over all cuts, so we expect agreement at $k=10\%$ and value only where the cut matters.

Why not just use variance or Shannon entropy? Pixel entropy describes uncertainty at one location and does not say how uncertainty is shared across cases. Variance across cases depends on scale and is dominated by single extreme cases, which makes cross-task comparison hard. These are reasons for preference, not proof, which is why the coefficient-of-variation ablation is included. If it reproduces every conclusion, Gini is a convenience rather than a necessity.

Why not just train against the averaged human soft map? Averaging is a training choice, and our proposal is an evaluation. Averaged-target models such as DML are among the methods we score. The open question is whether their good calibration on Prostate T1 coincides with correct allocation across cases. Calibration error averaged over pixels cannot tell.

Proposed Experiments

We will run all method families on the six QUBIQ tasks, LIDC-IDRI (four radiologists per nodule), Gleason 2019 (six pathologists) with Arvaniti TMA as an external test, and the 25-rater breast cancer set. LIDC-IDRI and the histopathology sets provide large case counts that stabilize the Gini estimates. QUBIQ provides modality and organ diversity.

The first decisive check is descriptive: whether human disagreement Gini is high. If it is near zero, so that disagreement is spread evenly across cases, the premise fails and the project stops.

The second is the triage simulation. For each model and each review budget $k$ from 5% to 30%, we flag the top-$k$ cases by model uncertainty and measure the share of total human disagreement captured. We then test whether the alignment ratio or Spearman better predicts this yield across methods.

The third is the reordering test. Does any pair of methods with overlapping mean Q-Dice or GED intervals separate under the concentration gap with non-overlapping bootstrap intervals? Specifically, does the ensemble's QUBIQ advantage persist on the high-disagreement tail?

The fourth is the intervention probe. We oversample by disagreement and report whether the concentration gap closes without lowering mean Q-Dice. A negative result here would extend the re-weighting failure documented for code models to multi-rater medical segmentation.