Where Do Annotators Disagree? Measuring and Training Against the Agreement Ceiling One Pyramid Level at a Time

When several experts outline the same skin lesion, tumor, or optic cup, they usually agree on whether something is there and roughly where it sits. They disagree about exactly where its edge runs. A single "agreement ceiling" number, such as a mean inter-rater Dice score or Cohen's kappa, mixes these two kinds of disagreement together. A model's score against such a ceiling therefore cannot tell us whether a new method got better at the part humans agree on, or just learned the boundary habits of whoever labeled the test set. We propose to take the agreement ceiling apart by spatial scale, using the same grid that spatial pyramid pooling uses. Spatial pyramid pooling is a network layer that summarizes a feature map over a 1×1, 2×2, 4×4 (and finer) grid of cells. Here we apply that grid to the raters' masks. For each grid level we compute how much raters disagree about the fraction of each cell that is foreground. This yields an agreement curve over scale rather than one number. We then compare models against humans level by level.

The same decomposition gives a training rule. If human disagreement at a level is measured, a model trained on several raters' masks can be told to match the raters' average closely at levels where raters agree. At levels where they scatter, the constraint is loosened in proportion to the measured scatter. The core claim is twofold, and both parts can be tested. First, on standard multi-rater medical segmentation datasets, disagreement grows sharply from coarse to fine pyramid levels, so the familiar ceiling is mostly a fine-scale phenomenon. Second, current models are often below human agreement at coarse levels while already matching or "beating" it at fine levels. If so, the useful headroom lies where the single-number ceiling hides it.

Core Claim and Relation to Prior Work

Several papers motivate the claim that the ceiling is real and binds. ISLES 2016 and 2017 reports neuroradiologist inter-rater Dice of 0.58 against winning model Dice of 0.30–0.32. Label Convergence bounds LVIS detection between 62.64 and 67.52 mAP and shows top detectors at 66.8. MRIQC argues that higher-capacity models overfit rater preferences instead of crossing the ceiling. All of these treat the ceiling as a single scalar. The closest hint at scale structure comes from Handling Inter-Annotator Agreement for Automated Skin Lesion Segmentation, which smooths ISIC rater masks with morphological operations and convex hulls before measuring agreement (median kappa ≈ 0.72). That paper found smoothing tightens consensus while bounding boxes worsen it. This is indirect evidence that disagreement lives at intermediate-to-fine scales, not at the coarsest one. It still modifies the labels rather than measuring agreement as a function of scale, and it does not change training. The other close hint is the Expertness Generator in DiFF. It suppresses high-frequency noise in learned rater-weight maps with a small CNN, which assumes rather than measures that disagreement is high-frequency.

Two lines of work supply method rather than motivation. Spatial Pyramid Pooling supplies the fixed 1×1/2×2/4×4 grid that we reuse as a measuring device. SPAM pooling and HSPP show that pooling over pyramid cells changes where gradients flow; for example, SPAM reaches Extent mAP 95.8 against 69.5 for global max pooling on MNIST128. Both were tested only on single-label benchmarks. Rater-modeling methods constrain the design. Pionono represents each rater by one global Gaussian latent vector, and multi-rater prompting uses rater prompt tokens in skip connections. Both model who disagrees but treat that disagreement as uniform across scale. Our proposal does not model individual raters at all. It models at what scale raters disagree, and it can sit on top of either approach. The negative result in Learn like a Pathologist is relevant too: adding 4/7-agreement patches hurt after a curriculum lifted AUC from 83.7% to 88.2%. Whole-image inclusion of ambiguous data can be destructive, and this motivates using ambiguous images for their coarse content only.

The Measurement

Let $y_r$ be rater $r$'s binary mask on an image. Let $p_r^{l,b}$ be the fraction of foreground pixels of $y_r$ inside cell $b$ of the $2^l \times 2^l$ grid at level $l$. Level 0 is the whole image; we use $l = 0,\dots,5$. Define the human disagreement at level $l$ as

$$D_l = \frac{1}{|P|\,B_l}\sum_{(r,s)\in P}\sum_{b=1}^{B_l} \left| p_r^{l,b} - p_s^{l,b} \right|,$$

where $P$ is the set of rater pairs and $B_l = 4^l$ is the number of cells. The same quantity computed between a model's soft output and each held-out rater gives the model's disagreement $\hat D_l$. The scale-resolved gap $G_l = \hat D_l - D_l$ is our evaluation object. $G_l > 0$ means headroom remains at that scale. $G_l$ well below zero at a fine level, on a test set labeled by the same raters as training, is a warning sign of rater-specific fitting. We check that warning by evaluating against raters held out from training. Bins are aligned to the lesion bounding region in addition to the full image, so that $D_l$ is not dominated by empty background cells. We report both alignments, because the "fixed grid rigidity" problem of standard pyramids would otherwise show up as an artifact.

The Training Rule

The model is an ordinary segmentation network. During training, its soft output $q$ is pyramid-pooled exactly as above and compared to the rater-mean occupancy $\bar p^{l,b}$:

$$\mathcal{L} = \mathcal{L}_{\text{pixel}} + \lambda \sum_{l} \frac{1}{\sigma_l^2 + \epsilon} \sum_{b} \left( q^{l,b} - \bar p^{l,b} \right)^2 .$$

Here $\sigma_l^2$ is the across-rater variance of occupancy at level $l$, estimated per image and shrunk toward its dataset average. $\epsilon$ is a small constant and $\lambda$ trades off the pyramid term against the usual pixel loss. The inverse-variance weights follow from treating each level's rater mean as a Gaussian estimate with that variance. Levels where raters agree get tight constraints, and levels where they scatter get loose ones. For images whose fine-level variance is extreme, the analogue of the 4/7 patches in the colorectal study, the fine-level terms and the pixel loss inside high-variance cells are down-weighted. The coarse terms are kept. This is a spatial, per-scale version of the curriculum's exclusion rule.

Research Program

The first phase is purely descriptive. We compute $D_l$ curves on every multi-rater dataset in reach: ISIC multi-mask images, RIGA (six glaucoma specialists), Gleason 2019 (six pathologists) and the 25-rater breast set used by Pionono, the BraTS-METS 2025 Lighthouse subset (challenge analysis), and MultiOrg. Lighthouse matters because it has two raters each annotating twice, which lets us split $D_l$ into inter-rater and intra-rater components per level. MultiOrg's $t^0/t^1$ rounds, where experts later ignored tiny structures, are the natural test of whether intra-rater drift is fine-scale. The second phase computes $G_l$ for published-style models trained under standard protocols, to find where the remaining headroom sits. The third phase tests the training rule against the baselines below.

Baselines

The naive reference is a U-Net trained with per-pixel loss on the majority-vote mask. This is what most pipelines do; it fixes the floor that every other method must clear.

The strongest simple competitor is the same network trained with per-pixel soft targets, the rater-mean mask. Averaging the pooled rater masks equals pooling the averaged mask, so this baseline already contains the information our pyramid term uses. If the pyramid term helps, it must be because of the per-level inverse-variance weighting and the selective down-weighting, not because of extra information.

The strongest published method is Pionono, which reports unweighted Cohen's kappa of 0.758 on Gleason 2019 (against 0.750 for the prior best) and 0.711 on the 25-rater breast data (Pionono). We will reproduce it on its backbone and data splits and add our pyramid term to it. Beating it means exceeding 0.758 kappa on Gleason 2019, or matching it while showing a smaller coarse-level gap $G_0$–$G_2$. On RIGA, the comparison is multi-rater prompting; for its segmentation accuracy, no reported result was found on Lacuna for PU-Net on RIGA, so we report our reproduction only. For ISIC, the file gives agreement statistics (median kappa ≈ 0.72) but no model segmentation number, so we treat that kappa as a human reference, not a model baseline.

The key ablation replaces the inverse-variance weights with uniform weights across levels. This is plain multi-scale deep supervision. It isolates whether measured disagreement, rather than multi-scale supervision itself, does the work.

All variants share a backbone, data splits, augmentation, epoch count, and an equal hyperparameter search budget of the same number of trials over learning rate and the single extra coefficient $\lambda$ or its analogue. Evaluation is always against raters held out from training where the dataset has enough raters.

Why not just…?

Why not just train on averaged soft masks? This is the real threat, because soft per-pixel targets already encode the raters' mean. It does not encode how reliable that mean is at each scale. With soft targets, a confidently shared coarse layout and a scattered fine boundary are weighted the same per pixel. The soft-target baseline is run head to head. If it matches the full method on both overall kappa/Dice and per-level gaps, we will report that the contribution is the measurement and not the loss, and the training rule will be dropped.

Why not just smooth the labels morphologically? Ribeiro et al. show that opening/closing and convex hulls raise agreement but leave a long tail of severe disagreements unresolved. Smoothing also removes fine structure for all images, including those where raters agree on it. We include convex-hull-conditioned training as an extra baseline on ISIC. Our per-level curves will show whether smoothing helps only at the levels where $D_l$ was high.

Why not just model each rater, as Pionono does? Rater models explain who disagrees and can generate rater-specific masks. They cannot by themselves say whether the consensus model has headroom at coarse scales. The two approaches are complementary, which is why the main experiment adds the pyramid term to Pionono rather than replacing it. If Pionono alone already closes the coarse-level gaps, that is a finding about rater modeling that our measurement makes visible.

Why not just use a bigger backbone? The ISLES plateau and the LVIS result in Label Convergence suggest capacity does not cross the ceiling. We test this directly by comparing per-level gaps for a small and a large backbone. Our prediction is that extra capacity shrinks fine-level gaps toward or past zero, which signals rater fitting, while leaving coarse gaps unchanged.

Proposed Experiments and Decisive Checks

The first decisive check is whether $D_l$ actually rises monotonically with level on ISIC, RIGA, Gleason 2019, and Lighthouse. It must also rise more steeply than a null model in which each rater's mask is randomly shifted by a few pixels; that null tells us how much of the rise pure boundary jitter would produce. If disagreement is flat across scale, as might happen for Gleason grade confusion between whole regions, the decomposition offers no leverage there, and we will say so per dataset.

The second check is whether trained baselines show positive coarse gaps alongside near-zero fine gaps. Agreement is measured with pixel-level kappa and Dice against held-out raters, the per-level gaps $G_l$, and Lighthouse's intra-rater split.

The third check is the training comparison above, with three seeds and paired bootstrap intervals. Success requires that the inverse-variance version beat both the soft-target baseline and uniform deep supervision on coarse-level gaps, without losing pixel-level kappa.

A final transfer test asks whether a model's coarse-level improvements, but not its fine-level ones, persist when it is scored against raters never seen in training. This is the property that would let the measurement separate real progress from fitting the test set's annotators.