How Much Dice Can Disagreeing Experts Allow? A Closed-Form Observer Ceiling for Medical Segmentation

When radiologists outline the same tumor or organ on a scan, their outlines differ. An automatic segmentation model is then scored by its overlap with one of these outlines, or with a vote over several of them. Overlap is usually measured with Dice, the fraction of voxels the two masks share, rescaled to lie between 0 and 1. Because the reference itself is noisy, even a perfect model cannot reach a Dice of 1. Nobody can currently say how far below 1 the ceiling sits for a given organ, a given panel of raters, or a given clinical pass threshold without collecting many more annotations. This note proposes a short analytical model of that ceiling. Each rater's outline is treated as the true boundary pushed in or out by a small random displacement. From that assumption, the expected Dice and the probability that a case falls below a clinical threshold follow in closed form from three quantities: how large the boundary displacement is, the ratio of surface area to volume of the structure, and the effective number of independent raters in the reference.

The template comes from wireless engineering. There, A New Look at Dual-Hop Relaying: Performance Limits with Hardware Impairments models transceiver imperfections as distortion noise that grows with signal power. It then proves that the signal-to-noise-and-distortion ratio approaches a finite ceiling, so adding power stops helping. We argue that adding raters to a reference behaves the same way, because raters trained under the same protocol make correlated errors. The model is useful to the extent that a small amount of data predicts a lot. Two or three raters per case should be enough to predict agreement with larger held-out panels, outage rates at thresholds nobody tuned for, and the per-organ ordering of Dice scores in challenges where only one annotator was available.

Core Claim

Let a rater's contour on case $c$ differ from the population-mean contour by a signed normal displacement. That displacement has a rater-specific offset, which plays the role of the per-rater mean in Pionono, and a session-level random part. Their combined spread is $\sigma_c$, measured in millimetres. For displacements small relative to the structure, the symmetric-difference volume is approximately the surface area $S_c$ times the mean absolute displacement. Consider an ideal predictor of the population-mean contour, scored against a consensus of $M$ raters whose errors have pairwise correlation $\rho$. Its expected Dice is then

$$\mathbb{E}[\text{Dice}_c] \approx 1 - \sqrt{\tfrac{2}{\pi}}\,\frac{S_c}{2V_c}\,\frac{\sigma_c}{\sqrt{M_{\text{eff}}}}, \qquad M_{\text{eff}} = \frac{M}{1+(M-1)\rho},$$

where $V_c$ is the structure volume. The factor $\sqrt{2/\pi}$ is the mean absolute value of a unit Gaussian, and $M_{\text{eff}}$ is the usual design-effect correction for correlated averaging. As $M\to\infty$, $M_{\text{eff}}\to 1/\rho$. The reference therefore stops improving and a ceiling remains, which is the direct analogue of the distortion ceiling in the relaying paper.

Ambiguity varies across cases, so we let $\sigma_c$ follow a Gamma distribution with shape $a$ and scale $\theta$. Given the clinical threshold $\gamma_{\text{th}}$, the probability that case $c$ "fails" becomes a Gamma survival function evaluated at a geometry-dependent point:

$$P_{\text{out},c}(\gamma_{\text{th}}) = \Pr\!\left[\sigma_c > (1-\gamma_{\text{th}})\,\frac{2V_c}{S_c}\sqrt{\frac{\pi M_{\text{eff}}}{2}}\right].$$

Cohort-level outage is the average of this survival function over cases. Case geometry is read from the masks themselves, and only $a$, $\theta$ and $\rho$ are fitted. This mirrors how Nakagami-$m$ fading turns into outage formulas in the communications literature.

What is new is a falsifiable formula linking rater disagreement, structure geometry and panel size to achievable overlap. It has three or fewer free parameters, and every prediction it makes can be checked against data it was not fitted on.

Relation to Prior Work

The analytical side supplies the method. A Simple Model of Inference Scaling Laws puts a Beta distribution over per-problem failure probability and derives pass@$k$ exactly. It also introduces an effective trial count $k_{\text{eff}}$ to absorb correlation between attempts. We borrow both moves: a distribution over case difficulty, here a Gamma over $\sigma_c$, and an effective count, $M_{\text{eff}}$. Two things differ. Our metric is a continuous geometric overlap rather than binary success, and our setting has no oracle verifier. The reference is itself the noisy object being modeled, which is precisely the open problem that paper flags.

The relaying paper supplies the ceiling structure. Its result on maximum tolerable impairment motivates our question: is there a level of rater correlation $\rho$ beyond which no panel size can push a given structure past a Dice target?

On the segmentation side, three related works motivate the proposal but do not overlap with it. A Probabilistic U-Net for Segmentation of Ambiguous Images learns a latent distribution over whole segmentations. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater a Gaussian whose mean is that rater's bias and whose covariance is that rater's inconsistency. The Latent Doctor Model for Modeling Inter-Observer Variability encodes doctor styles from a support set of that doctor's annotations. All three are estimators of the rater distribution. None says what overlap is achievable given that distribution, and none says how scores change with panel size; each requires training and Monte Carlo sampling to answer such questions. Our offset-plus-noise decomposition is deliberately the scalar, boundary-normal shadow of Pionono's latent mean and covariance.

QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge reports that simple ensembles often match sophisticated probabilistic models. That finding constrains us. If rankings between methods sit inside the observer ceiling's noise band, the ceiling should reveal it, and we treat this as a prediction to check.

One piece of prior work invites the geometric claim without explaining it. Segthor: Segmentation of Thoracic Organs at Risk in CT Images reports Dice of 0.93 for the heart and 0.80 for the esophagus. Our formula predicts exactly this ordering if raters disagree by a similar number of millimetres on every organ. The esophagus is thin, so its surface-to-volume ratio is several times the heart's, and the same displacement costs it far more Dice.

Research Program

The work has three parts.

The first is derivation. We will derive the linearized ceiling and an exact correction for small structures, using concentric spheres for which Dice has a closed form in radius and displacement. This matters because LIDC-IDRI nodules can be only a few millimetres across, where linearization breaks. We will also derive the Gamma-mixture outage formula and the large-$M$ limit.

The second is estimation. The parameters $a$, $\theta$ and $\rho$ will be fitted from as few as two or three raters per case, using signed distances between rater boundaries. We will show that correlation between raters is identifiable only with at least three raters, or with repeated sessions from the same rater. Error shared by every rater cannot be identified from rater data at all. We will state plainly that the model therefore bounds agreement with the expert population, not with anatomical truth.

The third is use. We will rescore published models on a headroom scale, defined as the model's Dice divided by the predicted ceiling for the same reference. We will then ask whether the method rankings in the multi-rater benchmarks survive that rescaling.

Baselines

The naive reference point is current practice. One reports raw Dice and, at most, the mean pairwise inter-rater Dice as an informal ceiling. This is computed on the same cases and raters. Our model beats it if it predicts agreement against panel sizes or thresholds that the pairwise number cannot address, while matching it within its own sampling error where both apply.

The strongest nonparametric competitor is bootstrap resampling over raters. For each available panel size, we would draw sub-panels, compute model-versus-consensus Dice, and fit a saturating power law in $M$ to extrapolate. It is the strongest competitor because it makes no geometric or Gaussian assumption. To keep the comparison fair, it will receive exactly the same fitting subset of raters and cases and will be evaluated on the same held-out raters. Beating it means a lower mean absolute error on held-out agreement once the extrapolation goes beyond the fitted panel size.

The strongest learned competitor is simulated raters. A Pionono model is trained on the fitting raters, and its rater latents are then sampled to synthesize larger panels. It uses the same training raters and is given the standard training budget from its paper. We compare with it because it captures spatially varying disagreement that our scalar $\sigma_c$ cannot. Beating it means comparable or better held-out prediction with no network training. Failing to beat it would mean the geometry-only model is too coarse.

For context, these are the model scores that will be rescored on the headroom scale. On QUBIQ brain growth, Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures reports continuous Dice of 0.5372, against 0.5042 for ESULR and 0.5083 for Bayes dropout. The QUBIQ paper reports top teams at average Dice between 0.80 and 0.85. Pionono reports unweighted Cohen's kappa of 0.711 on the 25-rater breast cancer data. For our actual task, predicting held-out rater agreement or outage from a small panel, no reported result was found on Lacuna, so the bootstrap and simulation baselines will be run by us.

Three ablations isolate the ingredients. The first fixes $\sigma_c$ to a constant, removing the Gamma ambiguity distribution; this should break outage calibration while leaving mean Dice roughly intact. The second sets $\rho=0$ so that $M_{\text{eff}}=M$; this should overpredict gains from larger panels. The third replaces per-case $S_c/V_c$ with a dataset-wide constant; this should destroy cross-structure prediction.

Why not just…?

Why not just use the measured inter-rater Dice as the ceiling? On a single dataset, at a single panel size, it is a fine number, and in that regime our model must agree with it. It cannot say what happens with five raters when you have two. It cannot give the fraction of cases below a clinical threshold. It also cannot transfer from a large organ to a thin one. Those three predictions are exactly what the held-out experiments test.

Why not just bootstrap over raters and extrapolate? This is a real threat. Within the range of observed panel sizes, a bootstrap will likely match us, and an empirically fitted power law may extrapolate passably. The decisive comparison is the 25-rater breast cancer data, where both methods are fitted on three raters and must predict agreement against panels of up to 22 held-out raters. If the bootstrap matches us there too, the contribution of the proposal shrinks to the geometric transfer and the closed-form outage parts, and we will say so.

Why not just simulate raters from a trained Pionono or Probabilistic U-Net? These models do represent disagreement richly. However, their samples inherit the models' own fitting errors. Answering any new question also requires retraining or resampling, which in 3D is the cost the QUBIQ paper identifies as a barrier. Running them as a baseline shows whether their extra spatial detail buys predictive accuracy over a three-parameter formula.

Proposed Experiments

The first check is panel extrapolation. On LIDC-IDRI (4 raters), QUBIQ brain growth (7 raters) and the 25-rater breast histopathology set, we fit on two or three raters per case. We then predict mean Dice of a consensus-matching predictor against held-out sub-panels of every size and report mean absolute error against the observed values.

The second check is outage calibration. On all six QUBIQ tasks, we compare predicted and empirical fractions of cases with Dice below thresholds from 0.5 to 0.9, using a reliability-diagram style error.

The third check is geometric transfer. We fit a single millimetre-scale $\sigma$ on one QUBIQ structure and predict the per-organ Dice ordering on SegTHOR from the organs' measured surface-to-volume ratios. The test is whether the esophagus-versus-heart gap is reproduced without organ-specific tuning.

The fourth check is ranking stability. We rescore the reported QUBIQ and LIDC-IDRI methods on the headroom scale. We then test whether differences such as the 0.0289 continuous-Dice gap between the variational method and Bayes dropout exceed the ceiling's predicted case-level noise.

The proposal fails if extrapolation error is no better than the bootstrap's, or if SegTHOR ordering cannot be recovered with one shared displacement scale. Either result would show that rater disagreement is not well summarized by boundary displacement and geometry.