Separating Rater Disagreement from Registration Error in Propagated Labels

When a medical image segmentation model has only a handful of expert-annotated scans, a common shortcut is to copy those annotations onto unlabeled scans by spatial registration, which means computing a deformation that aligns one patient's anatomy to another's and warping the label along with it. Each copied label is then wrong in two unrelated ways. The registration may be misaligned, and the original expert may have traced the boundary differently from how another expert would. At soft, low-contrast boundaries both errors are largest, and they look the same: a boundary shifted a few millimetres. Current pipelines cannot tell which error they are looking at. That matters because the two need opposite fixes. Registration error shrinks if you add more donor scans or register better. Rater bias travels with the donor's label into every patient it is warped onto, and it only shrinks if you ask a second expert.

We propose to measure the two error sources separately. Every donor label is warped onto every target scan in a crossed design, and the resulting boundary offsets are fit with a variance-components model. In that model a rater's bias is shared across all targets a donor reaches, while registration error is specific to each donor–target pair. A small number of donors traced by two experts makes the two components identifiable. The fitted model does two things. It produces training targets whose boundary uncertainty reflects only genuine expert disagreement, and it gives a rule for spending a fixed expert-annotation budget, either on second opinions or on more donors.

Core Claim

Consider a set of donor scans, each traced by a known rater $r(d)$, and a set of target scans. Map every boundary point into a common template surface coordinate $u$. Let $o_{d,t}(u)$ be the signed distance along the surface normal between donor $d$'s label warped onto target $t$ and target $t$'s unknown true boundary. We model

$$o_{d,t}(u) = \beta_{r(d)}(u) + \gamma_d(u) + \epsilon_{d,t}(u) + \eta,$$

where $\beta_r$ is a smooth field of systematic bias for rater $r$ (for example, consistently tracing the myocardium outside the true wall where contrast is weak), $\gamma_d$ is donor-specific effort that persists across targets, $\epsilon_{d,t}$ is pair-specific registration error, and $\eta$ is pixel-level noise. The target's true boundary enters as a per-target intercept, so it is estimated jointly and never needs to be observed.

This is a two-way random-effects model of the kind used to separate subject and rater effects in reliability studies. Here it is placed on a surface and applied to propagated labels. One identification problem has to be stated honestly. Systematic registration failure for an atypical donor also appears in every target, so it is confounded with $\gamma_d$. This is the systematic ceiling $a$ in the convergence model of Multiclassifier Fusion in Human Brain MR Segmentation: Modelling Convergence, which motivates this work. We break the confound with a small anchor set of donors traced by two raters. On those donors the rater terms can be read directly, with no registration involved. The question of how many anchors are needed is itself one of the proposal's outcomes.

The claim is that this decomposition can be recovered accurately from existing low-data setups. We further claim it makes two testable predictions. First, rater variance concentrates where local image contrast is low, while registration variance concentrates where shape varies across subjects. Second, allocating annotation effort by the fitted components beats fixed allocation policies at equal expert time.

Relation to Prior Work

Four lines of work sit next to this one, and none of them separates the two error sources.

Registration-based propagation is our closest relative and supplies the propagation machinery. Learning Semi-Supervised Medical Image Segmentation from Spatial Registration (CCT-R) warps dense labels from very few annotated volumes and filters bad pairs by cycle consistency. It treats donor labels as clean truth, so any rater bias is silently copied into every target. We reuse its propagation and pair-filtering steps. What we add is an explicit rater term.

Registration-uncertainty work motivates the problem. From Registration Uncertainty to Segmentation Uncertainty shows that image-space registration quality barely predicts label error (Pearson $r<0.10$), whereas label-space uncertainty reaches $r=0.583$ on ACDC and M&Ms. However, it assumes a single correct moving label and uses uncertainty only after the fact. Our model asks how much of that label-space uncertainty is actually rater disagreement.

Label-fusion rater models provide the statistical ancestry. STAPLE-style fusion and Non-local STAPLE: An Intensity-Driven Multi-atlas Rater Model treat each warped atlas as a "rater" with a performance level. Non-local STAPLE relaxes strict voxel correspondence through patch search and gains a median 0.08 Dice over STAPLE on low-contrast thyroid CT. In these methods, however, the "rater" is the atlas: human tracing bias and registration error are lumped into one reliability number per atlas.

Deep annotator models do the reverse. Disentangling Human Error from Ground Truth in Segmentation of Medical Images learns image-conditioned, per-pixel confusion matrices for each annotator and improves confusion-matrix estimation over STAPLE by 14.4% on BraTS. Because it works within a single image's coordinates, applying it to warped labels would absorb registration error into the annotator term. Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle (DiFF) reweights raters by downstream diagnostic value but has no cross-subject spatial model.

Finally, Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment constrains the design. It finds that three annotators capture most of the benefit of consensus (model F1 IQR drops from 0.038 to 0.016, with only +0.013 F1 from going to eleven). This suggests that a few doubly-annotated anchors, not exhaustive multi-rating, should be enough to estimate rater fields.

Research Program

The first component is the estimator. Using CCT-R-style registration (affine then deformable, with cycle-consistency filtering), we warp every donor onto every target and extract signed normal offsets on a template surface for each structure. We then fit the model by restricted maximum likelihood, with each field expanded in a low-dimensional smooth surface basis. Output is the posterior mean of each target's true boundary and three variance maps over the surface: rater, donor, and registration.

We also add an image-evidence term to help identifiability. A rater's bias is visible in the donor's own image, as the offset between the traced boundary and the donor's own intensity edges, without any registration. Registration error is visible as residual mismatch between the warped donor image and the target image. These enter as weak covariates, not as hard constraints, because at low contrast the intensity edge is itself unreliable.

The second component is training. A segmentation network is trained on unlabeled targets using soft boundary bands centred on the estimated latent boundary. Each band's width is set by the rater variance at that location only, since registration variance has been averaged out by the crossed design. The intended effect is that boundary losses no longer push the network toward confident edges where experts genuinely disagree. The file flags this as a risk of overlap-style losses such as those in Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation. At the same time, the bands avoid being blurry where disagreement is merely registration noise.

The third component is allocation. Given the fitted variances, we compute the expected reduction in latent-boundary error from one more donor versus one more second opinion on an existing donor. We then spend a fixed budget of expert time greedily according to that estimate.

Baselines

The naive reference point is single-donor propagation: one labeled scan, registered and warped, then used to train a segmenter. CCT-R is the strongest version of this with dense donors. It reports 80.4% mean Dice on ACDC with one labeled case (versus 58.6% for MCSC) and 47.6% on Synapse. We will rerun CCT-R with its published registration settings on our exact splits and treat those figures as a sanity check. Beating it means higher Dice and lower 95th-percentile Hausdorff distance at the same number of expert-traced scans. The comparison uses the same donors and the same registration outputs, and the extra registrations our crossed design needs are given to every baseline that can use them.

The strongest classical competitor is multi-atlas fusion over the same crossed warps, run as majority vote, STAPLE, and Non-local STAPLE. Heckemann et al. report that fusion raises similarity index from 0.754 with one atlas to 0.836 with 29 on healthy brain MRI. These methods receive exactly the same warped labels as our model. They must be beaten on latent-boundary error and on calibration of boundary uncertainty, not only on Dice.

The strongest deep competitor is the confusion-matrix model of Zhang et al., trained on warped labels with donor-rater IDs. It is given the same tuning budget as our model: an equal number of hyperparameter trials selected on the same validation split. We also include the dual-uncertainty TransMorph model as the competitor for the diagnostic task of predicting where propagated labels are wrong.

Two ablations isolate the key ingredient. One drops the rater terms, so all variance is attributed to registration. The other drops the pair term, so all variance is attributed to raters. A third ablation removes the anchor donors to show the confound directly.

Why not just…?

Why not just fuse many warped atlases with STAPLE and train nnU-Net? When donors come from several raters, fusion does average rater bias together with registration error, and it may match our Dice. We take this as a genuine threat. Fusion cannot say which error remains, however, so it cannot guide the next annotation. It also fails when most donors share one rater, because that rater's bias does not average out. The allocation experiment and a single-dominant-rater split settle whether the decomposition is worth anything beyond fusion.

Why not just apply Zhang et al.'s annotator confusion matrices to the warped labels? This is the most plausible drop-in alternative. Its confusion matrices are free to explain registration error as rater error, which should show up as inflated, patient-varying "rater" maps. If it nevertheless matches our segmentation accuracy, we will concede that the separation is not needed for accuracy. The contribution then narrows to the diagnostic maps and the allocation rule, which the pair-term ablation and the allocation experiment test directly.

Why not just have three experts trace one donor, following the "power of three"? Wilm et al.'s result makes this a sensible fixed policy. It is exactly one of the allocation policies we compare against at equal expert minutes, alongside "one tracing each on three donors." If it wins everywhere, the allocation rule is unnecessary.

Why not just use registration uncertainty to down-weight bad pixels? The TransMorph dual-uncertainty model correlates with error ($r=0.583$) but assigns rater-induced error to the image or registration terms. We test whether its uncertainty maps coincide with our rater maps or with our registration maps.

Proposed Experiments

The first experiment is controlled recovery. On ACDC and Synapse we treat the released masks as the reference and simulate raters with smooth signed offset fields whose magnitude grows as local contrast falls. We then measure how accurately each method recovers the rater and registration maps and the latent boundary, as a function of the number of anchor donors. Because this simulation matches our model's form, we also run misspecified raters: pixel-flip noise, topological slips, and a bias correlated with anatomical atypicality. The decisive check is whether recovery degrades gracefully and whether the anchor count needed plateaus around two to three, as Wilm et al.'s findings suggest.

The second experiment uses real disagreement on REFUGE-2. Its multi-rater optic disc and cup annotations sit on homologous anatomy with a notoriously soft cup boundary. We hold out one rater per image as a stand-in for an unseen expert and score how well each method's predicted boundary band covers that rater's tracing. No reported result found on Lacuna for REFUGE-2 optic-cup Dice under label propagation; DiFF reports only a +2.73% diagnostic AUC improvement there. We will also test the first prediction directly: rater variance should correlate with local contrast more strongly than registration variance does.

The third experiment compares allocation at a fixed expert budget on ACDC with simulated raters and on REFUGE-2 with real ones. The policies are single tracings on new donors, the three-raters-on-one-donor policy, and our variance-guided greedy rule, each followed by the same segmenter training. We report Dice, 95th-percentile Hausdorff distance, and boundary-band calibration. The proposal fails if the variance-guided rule does not beat the better of the two fixed policies at any budget. It also fails if Zhang et al.'s model on warped labels produces maps just as predictive of where a second opinion helps.