Who Labeled This? Separating Rater Style from Scanner Shift in Multi-Site Pathology

When a medical image classifier trained at one hospital loses accuracy at another, the usual explanation is that the images changed: a different scanner, a different stain batch, a different patient mix. A second cause gets less attention. The labels may also have changed. Each hospital's pathologists draw the line between, say, Gleason grade 3 and grade 4 prostate tissue in slightly different places, and the model is scored against those local opinions. A third problem sits underneath both. Models that try to learn each annotator's habits depend on knowing which annotator produced which label, and in pooled multi-site data that bookkeeping is often wrong. The same pathologist can appear under two IDs, two residents can share one, and IDs can be scrambled during de-identification. This is the same failure that splits one author's citation record when a surname is misspelled.

We propose a model of grading labels with three parts. An image encoder handles scanner and stain differences. A per-rater "style" component varies with tissue appearance, so a rater can be reliable on well-formed glands and unreliable on cribriform sheets. A third component treats the attached rater ID as noisy metadata and infers which IDs belong to the same underlying person. At a new hospital, the image model stays frozen and only the new raters' styles are estimated from a few labeled cases. The deliverables are a measurement and a method. The measurement splits the cross-site accuracy gap into an imaging part and an annotation part. The method holds up when rater identities are corrupted, and it makes "expert-level" claims answerable to a stated question: expert-level relative to whom.

Core Claim

Three bodies of work each solve one piece of this and assume away the others. The rater-modeling line assumes clean rater IDs and a single imaging distribution. The Latent Doctor Model encodes a clinician's style from a support set of their other labeled images, using a conditional variational autoencoder (a generator whose randomness is shaped by a learned latent code). Pionono gives each rater a Gaussian in latent space: its mean captures systematic bias and its variance captures self-inconsistency. SVGPCR on CrowdGleason learns one confusion matrix per annotator. The file names two limitations of this line explicitly: confusion matrices that do not depend on image features, and no coupling to cross-center domain shift. The domain-shift line, exemplified by targeted augmentations and Connect Later, treats target labels as objective truth.

Our claim is that these limitations interact, and that fixing them jointly is required. If rater style is not modeled, a model that ignores rater differences will blame annotation disagreement on the image encoder, and domain adaptation will then try to "fix" image features that were never the problem. If rater identity is not resolved, a rater-aware model will learn averaged or split styles that match no real person. The finding that institutional annotation practice is a primary driver of cross-site gaps (Gedon, described in the file without a linked paper) motivates the first half of this claim. The fragmentation pattern documented in author corrections motivates the second.

Why Author Corrections Are the Right Lens

The scholarly-correction records in the file are effectively a catalog of how identity metadata breaks.

They show single-character slips: "Wojciechowki" for Wojciechowski in Farkas Bounds on Horn Constraint Systems, and "Xinglin" for Xinlin in the black rockfish genomics correction. They show ordering and capitalization errors for non-Western names, in the data imputation erratum. They show misattribution inside a 49-author list on a chest X-ray deep learning paper, in the COVID-19 pneumonia pipeline correction.

We turn these into a taxonomy of corruptions on rater IDs. Splitting one rater into two IDs corresponds to a misspelling. Merging two raters into one ID corresponds to a name collision or ordering confusion. Swapping a block of labels between raters corresponds to misattribution within a large team.

There is a sharp contrast with the publishing case. The radiomics corrigendum emphasizes that an author correction changes nothing about the science. In a rater-aware model the opposite holds: a mislabeled annotator ID changes the learned weights. Because no journal erratum can repair that, the repair has to be done by inference.

Research Program

The model predicts a label $y$ for image $x$, given an observed rater tag $t$ and site $s$, as

$$p(y \mid x, t, s) = \sum_{r} \pi(r \mid t)\; p\big(y \mid f_\theta(x),\, z_r(f_\theta(x))\big).$$

The terms are as follows. $f_\theta$ is a shared image encoder trained with stain-space targeted augmentation in the style of Gao et al., using the two-stage schedule of Connect Later. $z_r(\cdot)$ is the style of true rater $r$. It is written as a function of the image features because it is a small mixture over learned morphology clusters, not one global confusion matrix. $\pi(r \mid t)$ is a sparse assignment from observed tags to true raters, initialized to the identity.

Tags whose inferred styles are statistically indistinguishable on overlapping cases become candidates to merge. A tag whose labels fit a two-component style much better than one becomes a candidate to split. This is the same evidence an author-disambiguation system would use, applied to labeling behavior instead of name strings.

Site enters in two ways. It enters through the augmentation, which addresses imaging. It also enters through a hierarchical prior that lets raters from the same institution share a site-level style mean, which addresses annotation culture.

At deployment the encoder is frozen, in keeping with the locked-predictor constraint that motivates INPUTADAPTER. New raters' styles are inferred from a small support set, as in the Latent Doctor Model. The output is a prediction of what an expert pool would say together with a forecast of how each local rater will grade.

The analysis has three parts. The first is a decomposition of the cross-site gap into imaging and annotation components. The second is a robustness curve under controlled ID corruption. The third is a re-examination of "expert-level" claims. Parity results such as CheXNeXt or the multi-expert fundus system are scored against particular reader pools. We will report model–rater agreement against the inter-rater agreement ceiling, separately for each rater pool.

Baselines

The naive reference point is a classifier trained on majority-vote labels, with standard augmentation and no rater information. This is the "consensus-driven training" the file identifies as the default for expert-level models. Its cross-site drop is the quantity we are trying to explain.

A second baseline adds stain-space targeted augmentation to majority-vote training. It isolates how much of the gap is fixable on the imaging side alone.

The strongest existing rater-aware competitors are SVGPCR and SVGPMIX from CrowdGleason and the conditional Latent Doctor Model. CrowdGleason is the natural benchmark because it pairs crowd and expert labels. No reported result was found on Lacuna for SVGPCR, SVGPMIX or LDM on CrowdGleason, so we will re-run all of them ourselves rather than quote numbers.

Fairness requires that every method share the same patch encoder backbone, the same stain augmentation, the same training splits and a matched hyperparameter search budget. Then the comparison isolates how labels are modeled. Beating these baselines means improving rater-specific label likelihood and kappa against held-out experts at the unseen site, and degrading more slowly under ID corruption. We set the threshold at improvement beyond the seed-to-seed spread of the baselines.

A per-site recalibration baseline fits a temperature and class thresholds on the same small target support set our method uses. Three ablations isolate our ingredients. The first removes identity resolution by fixing $\pi$ to the identity. The second removes feature dependence by using a single confusion matrix per rater. The third removes the site-level style prior.

Why not just…?

Why not just train on soft labels from the rater distribution? Soft labels keep disagreement but erase who disagreed. As a result they cannot forecast how a new site's pathologists will grade, and they cannot detect that two tags are the same person. They are included as a baseline. If they match our method on expert-agreement metrics at the unseen site, then the rater-specific machinery would only be justified by the provenance experiments, and we would say so.

Why not just recalibrate thresholds per site? This is the real threat. If annotation culture differs between sites only by a shift in where the grade cutoffs sit, a few target cases suffice to fix it, and our model adds nothing. The file's motivation points the other way. Disagreement concentrates on borderline morphology such as gastric regenerative change versus dysplasia (Shi et al.), and on poorly differentiated patterns, so rater error should depend on the tissue. The experiment settles this directly. We stratify the gain over recalibration by morphology cluster. A flat gain of zero would mean recalibration is enough, and we would drop the feature-dependent style component.

Why not just use Pionono or the Latent Doctor Model with the given IDs? Both condition on rater identity as given. Our corruption experiments measure exactly how much they lose when IDs are split, merged or swapped. If they lose little, identity resolution is unnecessary. We expect merges to hurt most, because a merged ID forces one style to represent two people.

Proposed Experiments

CrowdGleason is the primary testbed. It has multiple annotators of differing expertise, and its expert-versus-crowd split serves as a proxy for two annotation regimes. We hold out data by source where possible and add stain perturbations to emulate scanner shift.

The ID-corruption study injects splits, merges and swaps at rates from 0 to 30 percent. It reports, as a function of corruption rate, rater-specific negative log-likelihood, Cohen's kappa against each held-out expert, and assignment accuracy for $\pi$.

The decisive check for the gap decomposition requires labels for the same target images from both source-site and target-site raters. We will collect a small cross-read set, on the order of a few hundred patches graded by pathologists at two institutions. On this set the annotation component of the gap is directly observable: it is the score difference when the same predictions are graded against the two rater pools.

Secondary checks extend feature-dependent styles to segmentation on a multi-rater contouring task. There we compare against Pionono using generalized energy distance between predicted and observed contour sets.

The proposal succeeds if two things hold. First, a non-trivial share of the cross-site gap is annotation-driven. Second, modeling feature-dependent styles with resolved identities beats per-site recalibration on borderline morphology while staying robust to realistic ID corruption.