In medical imaging, the images from one hospital are often labeled by that hospital's own pathologist. When this happens, two very different things change together from site to site: the scanner that captured the pixels and the person who decided what the pixels mean. A model trained on such data can learn to tell which hospital an image came from by its blur or color, and then use that cue to guess how strict the local expert was. On a new hospital with an unfamiliar scanner, that guess no longer holds. Standard fixes for distribution shift make this worse. They try to make the model's internal features look the same across sites, but the sites really do differ in how often they say "positive," because the raters differ.
We propose to give the model each rater's identity explicitly, so it can learn rater strictness from that rather than inferring it from scanner cues. We then remove scanner information from the shared features only after accounting for rater and class, and predict at test time by averaging over the known raters. We will test this on a construction where the amount of rater–site entanglement can be dialed from none to total, which existing benchmarks do not allow.
Core Claim
The claim has three parts. First, when raters are nested within acquisition domains, both plain training and standard domain-adversarial training degrade on unseen scanners. The degradation should grow with the degree of nesting, and it should appear even though every rater is individually competent. Second, most of that loss is recovered by conditioning on rater identity and making the domain discriminator conditional on rater and class. The domain discriminator is the auxiliary network that tries to guess the site from the model's features. Third, a rater-conditioned model's per-rater outputs on an unseen scanner should still reproduce that rater's own labels. This last part is a direct test of whether learned rater style survives scanner shift, which the file flags as untested for latent rater models.
The objective, for detector or classifier $f$ with shared features $h(x)$, rater embedding $r$, domain $d$, observed label $y$, and discriminator $g$ trained through a gradient-reversal layer, is
$$ \min_{f,h}\max_{g}\; \mathbb{E}\big[\ell(f(h(x), r), y)\big] - \lambda\, \mathbb{E}\big[\ell_{d}(g(h(x), r, y), d)\big]. $$
Here $\ell$ is the task loss, $\ell_d$ is the domain-classification loss, and $\lambda$ weights the adversary. Because $g$ sees $r$ and $y$, it can only penalize scanner information that remains after rater and class are known. This is the class- and rater-conditional analogue of marginal alignment. At test time the prediction is the mean of $f(h(x), r)$ over training raters.
Relation to Prior Work
Four papers motivate the gap. Are We Modeling the Task or the Annotator? showed that text models can fingerprint crowd annotators, with recognition F1 up to 0.91, and lose up to 23 points on unseen annotators. It recommends disjoint-annotator splits but offers no training mechanism, and it never considers sensor shift co-occurring with annotator shift. The MIDOG challenge (Mitosis domain generalization in histopathology images - The MIDOG challenge) studies scanner shift but collapses raters into one consensus before training, as does its reference method. Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment quantifies rater variability, with individual F1 against consensus ranging from 0.64 to 0.85, but on a single acquisition setting. That paper also supplies our main testbed: 12 pathologists labeled every image, which is what lets us manufacture nesting at will.
Two papers supply the method. Domain Adversarial RetinaNet supplies the gradient-reversal detector we build on; we change only what the discriminator conditions on. The Latent Doctor Model shows rater style can be captured in a learned embedding. It assumes a fixed acquisition environment, and we test exactly whether such embeddings transfer across scanners. We use a simple embedding lookup rather than its variational encoder, because rater IDs are known here.
Two papers constrain the design. Direct-Effect Risk Minimization removes spurious domain-to-label paths and rebalances validation, but it assumes clean labels. In our setting, the "spurious" path from site to label is partly a real rater effect that must be modeled rather than deleted. LiNC learns per-sample trust weights that separate clean from noisy labels. Its authors note that it fails when errors are systematic, and rater-specific thresholds within a site are systematic by construction.
Research Program
The primary testbed is the canine mast cell tumor dataset from Wilm et al.: 50 images, 12 pathologists, 35/5/10 train/validation/test images, and a test gold standard of at least 7 of 12 votes.
We will create $k$ synthetic acquisition domains by applying fixed, scanner-like image transforms to disjoint groups of training images. MIDOG identifies the scanner differences that defeat color augmentation, namely contrast, sharpness, depth of field, and sensor noise, so the transforms will mimic those, for example a low-contrast blur in the style of the Leica GT450. Each group keeps only one pathologist's labels.
A nesting parameter $\rho \in [0,1]$ controls the assignment. At $\rho=0$ each image's labels come from a randomly drawn rater. At $\rho=1$ each domain is labeled entirely by its own rater. We will also include an adversarial assignment that pairs the strictest rater with the most degraded domain. Test images receive a held-out transform that was never seen in training, so evaluation is on an unseen scanner against a 12-rater consensus.
A second testbed repeats the construction for grading on Gleason2019, which has 6 pathologists and 4 classes. There we score both accuracy against the majority grade and KL divergence to the 6-rater label distribution.
Model selection uses leave-one-domain-out validation. Each held-out domain is scored against its own rater, and the rater-conditioned model uses that rater's embedding. This addresses the selection problem DRM raises, now with noisy single-rater validation labels.
Baselines
The naive reference is ERM: the same RetinaNet with a ResNet-18 backbone used by Wilm et al., trained on pooled single-rater labels with no domain handling. At $\rho=0$ with no transforms, this should reproduce the reported in-domain F1 of 0.74–0.76 for single-rater training. That number calibrates our pipeline.
The strong augmentation baseline uses the standard augmentations from the MIDOG reference paper, which reported test F1 0.6982 on MIDOG versus 0.6279 without normalization.
The strongest existing method is domain-adversarial RetinaNet with an unconditional discriminator. It reported 0.7183 test F1 on MIDOG's unseen scanners and 0.7339 on PANNORAMIC 1000 (Domain Adversarial RetinaNet). It is the right competitor because it is the published reference for exactly this detection task under scanner shift. We use it unchanged except for sharing our data splits.
We add LiNC trained on pooled labels as the strongest label-noise competitor. Its MedMNISTv2 results are 80.23% accuracy versus 58.82% for standard training under 50% symmetric noise, and noise-detection AUC 0.9837 on OrganSMNIST. Those results are under random rather than nested noise.
The upper reference is an oracle trained on 3-rater consensus for every image. Wilm et al. found that 3-rater consensus already reached about 0.75 F1 and that more raters added roughly 0.013.
Two ablations isolate the key ingredient. The first adds rater ID as an input to the adversarial model while keeping the discriminator unconditional. The second keeps rater conditioning but removes the adversary.
All methods share the backbone, training epochs, augmentation pool, and a fixed budget of hyperparameter trials selected by the same leave-one-domain-out protocol. Results are averaged over at least five rater-to-domain assignments per $\rho$. Beating the strongest baseline means a higher unseen-scanner F1 at $\rho \ge 0.5$ by more than the across-assignment spread, together with a gap that widens with $\rho$. There should be no loss at $\rho=0$.
Why not just…?
Why not just pay for three raters per image and train on consensus? Where budget allows, this is the right answer, and it is our oracle. But nesting is common precisely because consensus is expensive. Consensus also does not remove biases that raters share: in Re-thinking and Re-labeling LIDC-IDRI, multi-radiologist labels still produced 0.97 sensitivity and 0.23 specificity. We will also report how many doubly-labeled images per domain the baselines need to match our method. That gives practitioners the actual price.
Why not just add rater ID as an input feature, as Geva et al.'s experiment suggests? This is the real threat. It is the first ablation, and it may recover most of the gain. If it matches the full method, the conditional discriminator is unnecessary. The finding then becomes the simpler and still useful one: under nesting, condition on rater and do not align marginals. Our expectation that it falls short rests on one argument. An unconditional adversary forces equal feature distributions across domains whose positive rates genuinely differ, and that should distort features as $\rho$ grows.
Why not just clean labels first with LiNC, then run domain adaptation? LiNC assumes the network learns clean patterns before noise. Under nesting, one rater's strict threshold is consistent within a domain, so the network learns it as signal and trusts it. Meanwhile, images from the oddest domain will look like noise. The pooled-LiNC baseline tests this directly. We will also report its trust scores broken down by domain to check whether flagged "noise" concentrates in the most degraded domain.
Why not just augment hard enough that scanner cues vanish? If the model cannot see the scanner, it cannot infer the rater from it. Two facts limit this. MIDOG reports that blur and contrast differences survive standard augmentation. In addition, tissue content itself differs between image groups, so rater identity can leak through case features. The augmentation baseline at matched transforms measures how much it closes the gap.
Proposed Experiments
The decisive check is the curve of unseen-scanner F1 against $\rho$ for every method. The claim fails if ERM and the unconditional adversary do not degrade with $\rho$, or if our method's advantage does not grow with $\rho$.
The second check tests whether rater style transfers across scanners. On the unseen-transform test images, each rater-specific output should match that pathologist's own held-out annotations better than the averaged prediction does, with F1 measured against that rater. If per-rater fidelity collapses toward the average, the model is confusing scanner with rater.
The third check repeats the study on Gleason2019. The reported reference there is CLDM's KL of 0.461 to the 6-rater distribution in-domain, from The Latent Doctor Model. We do not know how that figure behaves under shift; no reported result was found on Lacuna for any rater-distribution model under scanner shift.
The final, cheap sanity check is a probe. A linear classifier on each model's features should be able to predict rater identity from images under ERM at high $\rho$, in the spirit of Geva et al., and should fail to predict it from our model's features once rater is supplied separately.
References
- Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets
- Mitosis domain generalization in histopathology images - The MIDOG challenge
- Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
- Domain Adversarial RetinaNet as a Reference Algorithm for the MItosis DOmain Generalization (MIDOG) Challenge
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Direct-Effect Risk Minimization for Domain Generalization
- LiNC: Lightweight Noise Correction via Adaptive Label Refinement
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction