In multi-hospital medical imaging datasets, two sources of non-biological variation are usually tangled together. Each hospital's scanner leaves a fingerprint on its images, and each hospital's radiologists have their own habits for drawing lesion borders. Because a given hospital's scans are almost always read by that hospital's own doctors, the two are perfectly confounded. A segmentation model trained on the pooled data can learn "images that look like Hospital A get drawn generously" without anyone noticing. At a new hospital it then reproduces the wrong convention, and its uncertainty estimates mix scanner degradation with real expert disagreement. Prior work has documented each effect separately. Networks amplify individual rater bias: a probe identifies the training rater from network outputs more reliably than from the raw labels (The Impact of an Inter-rater Bias on Neural Network Training). Scanner origin remains decodable at about 97% accuracy even after careful preprocessing (Machine Learning with Multi-Site Imaging Data). We found no work that studies the two together.
This proposal starts from an identification argument. When raters are nested within sites, no model, however expressive, can tell rater style apart from scanner effect using that data alone. Separation requires data that breaks the nesting. We propose to find out how little such data is needed and what kind. One kind is a small "bridge" set of scans read by raters from more than one site. The other is acquisition-paired augmentation, which re-renders the same anatomy as if it came from another scanner while keeping the label fixed. We also test a second, sharper claim: the standard fix of making the network's features scanner-invariant through an adversary is the wrong target when expert disagreement legitimately grows on low-quality scans.
Core Claim
Let $x$ be an image, $s$ its acquisition site, $r$ the rater, and $y$ the label. We want a model of $p(y \mid x, r)$ that transfers to unseen sites. Under full nesting, $r$ is a function of $s$, and $s$ is decodable from $x$. Any change in labels across raters is therefore equally well explained by a change across sites, and the training data cannot distinguish the two explanations. The claim has three parts.
First, a bridge set of crossed readings, combined with acquisition-paired augmentation, makes a rater-conditioned model recover each rater's true convention at held-out sites. In our semi-synthetic benchmark this ground truth is known.
Second, the two ingredients do different jobs. Augmentation stops the model from inferring the rater from the scanner fingerprint. The bridge set pins down rater style on real, rather than simulated, acquisition differences. Neither alone suffices once the real scanner signal falls outside what the augmentation simulates.
Third, full adversarial site invariance removes the image-quality information the model needs to widen its predicted disagreement on degraded scans. The result is miscalibrated uncertainty, even when average accuracy improves.
The novelty is not a new architecture. The proposal treats rater–site confounding as an identifiability problem with a measurable data requirement. It also separates two targets that differ: invariance to nominal site identity, and invariance to acquisition quality.
How This Relates to Prior Work
Two papers come closest. The Impact of an Inter-rater Bias on Neural Network Training trains U-Nets on each of two raters, using ISBI 2015 MS data and intracerebral hemorrhage CT. It shows significant cross-rater Dice drops, probe hit rates rising from 0.87–0.917 on raw labels to 0.90–0.93 on network outputs, and a systematic underestimate of lesion load by the junior-rater model. It pools scans and never models acquisition, so it motivates our work without addressing nesting. Scanner Invariant Multiple Sclerosis Lesion Segmentation from MRI attaches a scanner classifier to a 3D U-Net bottleneck. It uses a randomized cross-entropy confusion loss across 56 sites and raises Dice from 0.4540 to 0.5001, but against one deterministic mask per subject. It supplies our strongest invariance baseline, and its silence on rater variation is exactly the gap we test.
A second group models expert disagreement without asking where scans came from. OMBA-UNet uses one encoder with a Bayesian decoder per rater, and its rater-specific attention could be fitting scanner artifacts in scans assigned to particular raters. The Latent Doctor Model encodes a rater's past annotations into a style vector with a set encoder. We reuse this mechanism, because it scales to many raters, unlike one decoder per rater. Normalizing-flow probabilistic U-Nets (Improving Aleatoric Uncertainty Quantification…with Normalizing Flows) and mixtures of stochastic experts match disagreement distributions on LIDC-IDRI. All are evaluated on pooled splits. QUBIQ supplies our main distributional metric, Q-Dice (Dice averaged over nine thresholds of the predicted and reference rater-frequency maps). It also supplies a constraint: plain deterministic ensembles were competitive with probabilistic models, so any gain we claim must survive comparison to them.
Three papers act as constraints on design. Batch Effects In Brain Foundation Model Embeddings shows that post-hoc ComBat removes linear site signal from embeddings without improving diagnosis, which argues against fixing confounding after training. Detect and Correct Bias in Multi-Site Neuroimaging Datasets shows that harmonization must protect covariates or it erases biology. We read this as a warning that adversarial removal can erase legitimate signal, here quality-dependent ambiguity. Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment finds that three-rater consensus stabilizes models and that more raters add little. This hints that small crossed sets may suffice, a hypothesis we test directly with bridge-set size.
Research Program
The model is a single 3D U-Net encoder–decoder. The decoder is conditioned on a rater style vector produced by a Latent-Doctor-style set encoder, which reads a handful of that rater's past masks. Training combines three terms. The first is a per-rater segmentation loss on available readings. The second is an acquisition-consistency loss on paired augmentations: the same scan is re-rendered with simulated resolution loss, noise, bias field and contrast change, and the prediction for a fixed rater style must not change. The third is a bridge loss on the crossed subset, where one scan carries readings from raters of different sites.
Acquisition quality is not adversarially removed. The model sees a quality estimate (estimated noise and effective resolution) as an explicit input to a small head that predicts the spread of the rater distribution. The design intent is that nominal site identity carries no information about rater convention, while quality is allowed to shape uncertainty.
The program has three stages. Stage one is a measurement study answering a question the file shows nobody has asked: does inter-rater disagreement grow with acquisition degradation? On LIDC-IDRI, reconstruction kernels and slice thicknesses vary and up to four radiologists read each nodule. We regress pairwise rater disagreement on slice thickness and kernel, controlling for nodule size. If disagreement is flat across acquisition, part three of the claim is dropped and the work reduces to the identifiability study.
Stage two builds semi-synthetic nested benchmarks with known ground truth. In ISBI 2015 MS (Longitudinal multiple sclerosis lesion segmentation data resource), both raters labeled every scan on one 3T scanner. We create two or more synthetic "sites" through distinct, held-fixed acquisition transforms and show each site only one rater's labels. Because the withheld rater's labels exist, we can score whether the model recovers rater 2's convention on site-A images. We repeat the construction on LIDC-IDRI with four raters, using real acquisition groups, and on QUBIQ tasks with synthetic sites. The site transforms used to build the benchmark are kept disjoint from the augmentation family used in training. Otherwise augmentation would trivially undo the nesting.
Stage three sweeps the bridge fraction from 0% to 20% of scans, crossed with augmentation on or off. We then fit the curve of style-recovery error against bridge size.
Baselines
The naive reference is a single U-Net trained on all available labels, pooled with no rater or site information. It is the "do nothing" model that bakes the local convention into each site's predictions.
The first realistic reference is majority-vote or consensus training where multiple readings exist. Its known failure, in Re-thinking and Re-labeling LIDC-IDRI, is that consensus encodes shared bias: 0.97 sensitivity but 0.23 specificity for malignancy. We include it to show what collapsing raters costs under nesting.
The strongest existing competitor combines the two best-supported ideas in the file. It is the randomized cross-entropy scanner-confusion encoder from Scanner Invariant Multiple Sclerosis Lesion Segmentation from MRI feeding rater-conditioned decoding, in both OMBA-style per-rater decoder and Latent-Doctor style-vector variants. It is strongest because it directly removes site information from features while giving the decoder explicit rater identity, which in principle is everything needed. The only reported numbers for its invariance component are Dice 0.5001 against 0.4540 for the plain baseline on the in-house 56-site MS data. That dataset is unavailable, so the figure is context, not a target. We found no reported result on Lacuna for any method on a rater–site nested benchmark.
A deterministic ensemble of U-Nets, each trained on a rater subset, is included because QUBIQ found such ensembles competitive with probabilistic models. Top teams reached 0.80–0.85 on its staged Dice. A site-ID-conditioned model is also included.
All baselines share the backbone, preprocessing, data splits and bridge data where applicable. Each gets the same number of hyperparameter trials, with the adversarial weight swept for the invariance baseline, and the same GPU-hour budget. Ensembles receive matched total parameters.
Beating the strongest baseline means a statistically significant reduction in held-out-site style-recovery error across seeds. Style-recovery error is the Dice gap between the predicted and true mask for the withheld rater's convention. The win must come with no loss in pooled Q-Dice and with better calibration of predicted disagreement on degraded scans.
Ablations remove, one at a time, the bridge loss, the acquisition-consistency loss, and the quality input to the spread head. A further ablation replaces the quality input with adversarial removal, which isolates the third claim.
Why not just…?
Why not just make features scanner-invariant with an adversary and give the decoder the rater ID? This is the real threat. On semi-synthetic data, where rater labels were drawn on clean images, it may match our model on style recovery, and we will report that if it does. Our prediction is narrower. Under nesting, the adversary cannot help the decoder distinguish rater from site when bridge data is absent, because rater ID and site still co-vary in the labels. With bridge data, it should equal our model on mean accuracy but under-widen uncertainty on degraded LIDC-IDRI acquisitions. Stage one's measurement and the adversarial-removal ablation settle which is true.
Why not just train on majority-vote consensus labels? Under nesting, "consensus" at each site is that site's raters, so averaging cannot remove a site-level convention. The Impact of an Inter-rater Bias on Neural Network Training also shows networks amplify rather than average rater style. The consensus baseline measures this directly.
Why not just rely on heavy intensity augmentation or standard preprocessing? Site signal survives skull stripping, bias correction and whitening at 96.96–98.82% decodability (Machine Learning with Multi-Site Imaging Data), so simulated perturbations may not cover the real fingerprint. The "augmentation only, no bridge" arm tests exactly this. Our benchmarks hold the site transforms out of the augmentation family so the test is not rigged in augmentation's favor. If augmentation alone closes the gap, the bridge set is unnecessary and we say so.
Why not just ensemble rater-specific networks? Ensembles capture disagreement within the pool but have no mechanism to stop a member from keying on scanner fingerprint. They are run under matched parameters.
Decisive Checks
The first decisive check is the bridge curve on nested ISBI 2015 and LIDC-IDRI. If style-recovery error falls to the fully-crossed level with a bridge set of a few percent, the practical claim holds: consortia should invest in small cross-reading exercises. If it requires most scans to be crossed, the approach is impractical.
The second check uses the probe from The Impact of an Inter-rater Bias on Neural Network Training. A classifier tries to identify the training site from predictions made under a fixed rater style. Our model should push this toward chance where the baselines do not.
The third check uses LIDC-IDRI acquisition groups held out entirely. Predicted rater spread should track observed disagreement as slice thickness increases, measured by correlation and calibration error. The adversarial variant is expected to flatten this relationship.
The fourth check is clinical. Lesion-load estimates on nested ISBI data should not inherit the site's rater convention, the failure mode reported for the junior-rater model.
A null result on the first check would be informative too: it would mean nested multi-site labels cannot be repaired without fully crossed reading.
References
- The Impact of an Inter-rater Bias on Neural Network Training
- Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
- Scanner Invariant Multiple Sclerosis Lesion Segmentation from MRI
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Improving Aleatoric Uncertainty Quantification in Multi-annotated Medical Image Segmentation with Normalizing Flows
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Batch Effects In Brain Foundation Model Embeddings
- Detect and correct bias in multi-site neuroimaging datasets
- Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
- Longitudinal multiple sclerosis lesion segmentation data resource
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction