Doubly Transitive Annotation Designs for Learning Rater Disagreement

Medical images are ambiguous enough that different experts draw different outlines around the same tumor or organ. Models that learn this disagreement, rather than a single averaged mask, need several experts per image. Expert time is the scarcest resource in the pipeline, so in practice each image gets only a few of the available raters. Which few is usually decided ad hoc. We propose to choose these subsets using a structure borrowed from coding theory. Raters are assigned to images along the minimum-weight codewords of codes whose symmetry group is doubly transitive, meaning any ordered pair of positions can be moved to any other pair by a symmetry of the code. Under such an assignment every rater does the same amount of work, and every pair of raters co-annotates exactly the same number of images. No rater and no pair of raters is then left with thinner data than the others.

The idea comes from the proof that Reed-Muller codes achieve capacity on the erasure channel. There, double transitivity forces every pair of bit positions to lose information at the same rate, which turns a gradual degradation into a sharp threshold (Reed-Muller Codes Achieve Capacity on Erasure Channels). A partially annotated rater-by-image table is an erasure pattern: each missing annotation is an erased symbol that a rater-conditioned model must effectively infer. We claim that an erasure pattern with doubly transitive symmetry is the right way to spend a fixed annotation budget. Its effect should show up as uniform accuracy across raters and rater pairs, not only as a better average. The claim is testable today by subsampling datasets in which every rater labeled every image.

Core Claim

Fix $R$ raters, $n$ training images, and $k<R$ raters per image. A rater-conditioned disagreement model such as Pionono can then be trained on different allocations of the same budget. Pionono gives each rater a Gaussian in a learned latent space, whose mean captures that rater's systematic bias and whose variance captures their inconsistency (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation). We predict three things. First, allocations drawn from doubly transitive block structures will match random or cyclic allocations in mean accuracy. Second, they will cut the worst-rater and worst-pair error. Third, this advantage should be largest when $n$ is small, which is exactly the expensive-expert regime.

The design fact we rely on is that the codewords of a fixed weight in a code whose automorphism group acts doubly transitively form a balanced block design. Suppose $b$ blocks of size $k$ are drawn from $R$ points. Every pair of points then appears together in exactly

$$\lambda = \frac{b\,k(k-1)}{R(R-1)}$$

blocks. This is the pairwise analogue of each bit being equally exposed in Reed-Muller Codes: Theory and Algorithms. The balance follows purely from double transitivity, since the group maps any pair to any other while preserving the block set.

Parts of this are not new. Balanced incomplete block designs are classical in statistical experiment design, and their optimality for linear effect estimation is known. What is new here is threefold. We generate the designs from codes with doubly transitive symmetry, so they scale to realistic rater counts. We apply them to the nonlinear, spatial problem of learning rater-specific segmentation distributions. And we test the erasure-channel prediction that pair balance, rather than mere per-rater balance, is what flattens the error across raters.

Constructions Matched to Real Rater Counts

The rater counts in the available datasets happen to align with small symmetric codes. The QUBIQ brain-growth task has 7 raters per slice (QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge). Here the weight-3 codewords of the [7,4] Hamming code give the 7 lines of the Fano plane. The code's automorphism group acts doubly transitively on the 7 coordinates. The resulting design has $k=3$ and $\lambda=1$ per 7 images: cycling through the lines means every pair of raters meets once every 7 images.

For 8 raters, the weight-4 words of the first-order Reed-Muller code of length 8 give 14 blocks, which are balanced even on triples.

The Gleason 2019 prostate histopathology set has 6 pathologists. A 2-(6,3,2) design with 10 blocks exists and is preserved by a doubly transitive group on 6 points.

The 25-rater breast cancer set (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation) corresponds to the 30 lines of the affine plane over the field with 5 elements. That structure is invariant under the affine group, the same symmetry that makes affine-invariant codes capacity-achieving in Reed-Muller Codes Achieve Capacity on Erasure Channels. Each line has 5 raters, and every rater pair meets once.

Research Program

The work has three parts.

Simulation harness. On datasets where all raters annotated the same images, we hide annotations according to a chosen allocation. We then train a rater-conditioned model on what remains and score it on held-out test images that keep every rater. This turns a real annotation-protocol question into a controlled experiment, with no new labeling required.

Analysis of the erasure analogy. We define a per-pair recovery curve: how well the model predicts rater $j$'s mask on images where $j$ was hidden but $i$ was observed, as a function of the budget $nk$. This curve is the empirical counterpart of the EXIT function, the measure of how much information about an erased bit survives in the others. Under double transitivity these curves should coincide across pairs. Under transitive-only designs they should fan out.

The erasure-channel theory is asymptotic in blocklength, while our rater counts are single digits. We therefore do not expect a literal phase transition in $R$. The testable analogue is steepness in $n$: the budget at which the worst rater becomes well-modeled should arrive earlier and more abruptly under the symmetric design. If that does not happen, the analogy fails, and we will say so.

Transfer to partially overlapping data. For datasets whose existing overlap was not designed, such as the 25-rater breast set, we measure how far the observed incidence matrix is from pair-balanced. We then test whether imbalance predicts which raters Pionono models worst. That result would tell future annotation campaigns something even without new labeling.

Baselines

Full annotation. Every rater labels every image. This is the ceiling, not a competitor, since it costs $R/k$ times more. Pionono reports an unweighted Cohen's kappa of 0.758 on Gleason 2019 under its native annotation, against 0.750 for prior work (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation). The variational model of Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures reports a Continuous Dice of 0.5372 on QUBIQ brain growth, against 0.5083 for dropout and 0.5042 for ESULR. We will reproduce these numbers under full annotation first, so that all budgeted results are reported as a fraction of our own ceiling rather than compared to published figures from different pipelines.

Naive reference. Each image gets one randomly chosen rater, the common single-annotation practice, at the matched total budget.

Strongest competitor. Uniformly random $k$-subsets per image. For large $n$ these are nearly pair-balanced, so they are the real threat to our claim.

Adaptive allocation. Rater subsets are chosen per image to maximize Pionono's predictive disagreement on unlabeled images. It is given the same total budget and the same model.

Key ablation. A cyclic design of consecutive rater windows, in which each rater has equal load but pairs co-occur very unequally. Comparing it against the doubly transitive design isolates pair balance from per-rater balance. This mirrors the distinction between transitive and doubly transitive codes in the capacity proof.

All arms use the same backbone (Pionono's ResNet34 U-Net), training schedule, and hyperparameter search. The search is tuned once on full annotation and then frozen, with ten random seeds per arm. Random-subset arms also draw ten allocation seeds.

Beating the strongest competitor means one of two outcomes at a matched budget. The first is a statistically significant reduction in worst-rater personalized Dice error and worst-pair bias-estimation error, with mean Q-Dice within noise of random. The second is reaching 95% of the full-annotation ceiling at a smaller $n$.

Why not just…?

Why not just assign raters at random? This is the serious alternative. Random allocation is balanced in expectation, and its pair-count variance shrinks as $n$ grows, so at large budgets we expect the two to converge. Our claim is confined to the small-budget regime. There, with 7 raters and 3 per image over 70 images, a random allocation can leave some pairs co-annotating twice as often as others. The budget sweep across $n$ measures exactly where the gap closes. If the gap is negligible even at small $n$, the proposal reduces to "randomize and don't worry," which is itself a useful negative result for annotation planning.

Why not just use deep ensembles instead of rater-conditioned models? QUBIQ found that plain ensembles of deterministic networks often match or beat probabilistic models at reproducing expert disagreement, though at high cost in 3D (QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge). This objection is orthogonal to ours: ensembles are also trained on some allocation of expert labels. We will therefore also run a five-member ensemble trained on each allocation, scored with Q-Dice. We expect the allocation effect on the aggregate distribution to be smaller for ensembles than for per-rater personalization, and we will report that effect honestly.

Why not just choose raters actively where the model is unsure? Adaptive allocation can beat any fixed design when the model's uncertainty estimates are good. It is a baseline for that reason. It also has practical costs a fixed design avoids. Sequential rounds serialize the labeling, and early-round uncertainty is least reliable exactly when data are scarce. The small-$n$ comparison settles which effect dominates. A hybrid, a symmetric design for the first round followed by adaptive rounds, is a natural follow-up if adaptivity wins later in the sweep.

Proposed Experiments

Primary testbed: QUBIQ brain growth. All 7 raters label every slice, which lets us simulate Fano-plane, cyclic, random, adaptive, and single-rater allocations at $k=3$. We sweep $n$ from 14 to the full training set. The metrics are Continuous Dice and Q-Dice against the full rater distribution, per-rater personalized Dice, and the across-rater spread of that Dice. We also measure the error of each pairwise rater-bias estimate against its full-data value.

Second testbed: Gleason 2019. We use the 2-(6,3,2) design on the images annotated by all six pathologists, verifying that subset's size before running. The metric is Cohen's kappa, to connect with the published 0.758.

Observational check: 25-rater breast set. We regress per-rater modeling error on that rater's pair-co-occurrence imbalance in the existing labels. No reported result on Lacuna was found for allocation effects on any of these datasets, so all budgeted numbers will be new.

Decisive test. Everything turns on two comparisons at small $n$. The first is doubly transitive against random, where we must see a worst-rater advantage. The second is doubly transitive against cyclic, where the gap must exceed that between cyclic and random. If the design helps overall but cyclic performs as well, per-rater balance is doing the work, and the double-transitivity argument is unnecessary. If neither gap appears, the erasure analogy does not carry over to learned segmentation models.