Is the Cross-Hospital Gap Partly in the Labels? Blinded Central Re-reading as a Probe of Domain Shift

When a medical segmentation model trained at some hospitals loses accuracy at a new hospital, the loss is almost always blamed on the images: a different scanner, different noise, different patients. There is a second possible cause that nobody has measured. The ground-truth masks are usually drawn by each center's own clinicians. They know the clinical indication and follow local habits, for example about where a polyp's edge lies or how much of a trabeculated myocardium to include. If those habits differ between sites, then part of the measured "domain shift" is not the model failing on new images. It is the model being scored against a different reader's style.

Clinical research already has a tool for removing such habits: blinded independent central review (BICR). In BICR, trained readers who see only the image, with no clinical data, site, or scanner information, re-read every case under one structured protocol. We propose to use BICR as a measuring instrument for machine learning. On a subsample of existing multi-center benchmarks, we will re-read cases centrally and blinded, then score the same models against local labels and against central labels. The difference splits the cross-site gap into an image-shift part and a reader-shift part. We then test whether a model that separates site-specific annotation style from shared morphology closes the reader-shift part more cheaply than existing invariance methods do.

Core Claim

A measurable share of the cross-center performance drop reported on multi-center segmentation benchmarks comes from differences in local annotation, not from image appearance. That share can be estimated with a modest blinded re-read. Once estimated, it can be reduced by modeling annotation style explicitly rather than by forcing features to be site-invariant. The claim can fail cleanly: if models trained leave-one-center-out look just as bad against blinded central labels as against local ones, the gap really is in the images, and the field's current focus is justified.

What Prior Work Establishes and Leaves Open

The motivating evidence comes from clinical multi-center studies that relied on central blinded reading precisely because local interpretation could not be trusted to be uniform. The MRI-GENIE neuroradiological review (Diffusion-Weighted Imaging, MR Angiography, and Baseline Data in a Systematic Multicenter Analysis of 3,301 MRI Scans of Ischemic Stroke Patients) re-read stroke MRI from 12 sites with two neuroradiologists blinded to clinical and genetic data and reached 98% agreement on a validation subset. Frid et al. built phenotype associations on that harmonized reading. Neither study trained a model or asked how the harmonized labels change machine-learning generalization. That gap is ours.

A cautionary constraint comes from Stern et al., who blinded two readers to acquisition protocol when comparing standard and tin-filtered CT. They reported reader agreement of only κ = 0.65–0.67 and noted that the tin protocol's higher noise (41.5 vs 32 HU) could cue readers to the protocol. Blinding to site is therefore never perfect when images look different. Our protocol measures this leakage directly instead of assuming it away.

On the machine-learning side, the domain generalization literature surveyed in Domain Generalization for Medical Image Analysis: A Review organizes methods by data, feature, model, and analysis level. All of these levels treat labels as fixed truth. Benchmark papers such as TransNetR on six-center PolypGen, Double Encoder-Decoder Networks on unseen ETIS-LaribPolypDB, and AMOS, which reports MRI drops of more than 20% mDice on unseen scanners, report cross-center gaps without asking whether the reference masks share a reading convention. AMOS did use senior review of its masks, but it did not blind readers to clinical context or compare reading styles across sites.

The closest methodological neighbor is Benchmarking Dependence Measures to Prevent Shortcut Learning. That work removes a labeled confounder from learned features, and with a mutual-information penalty (MINE) it restores AUROC on a sex-confounded CheXpert split from 45.8 to 87.7. It targets confounders in the images. We target confounding in the labels, and we do not remove site information from features. Finally, ERM++ constrains us: it shows that well-tuned plain training beats specialized domain generalization methods on DomainBed, so every comparison here uses an ERM++-style tuned baseline.

Research Program

Blinded central re-reading. We take two public multi-center segmentation benchmarks where site is known and local annotation is plausible: PolypGen (colonoscopy, six centers) and the M&Ms multi-vendor cardiac MRI challenge. From each center we draw a stratified subsample, targeting on the order of a hundred frames per PolypGen center and a few dozen subjects per M&Ms vendor. Two central readers, blinded to site, clinical data, and the original mask, delineate every case under one written protocol. A third reader adjudicates disagreements, following standard BICR practice.

Three measurements accompany the re-read. First, the readers guess each case's site, which quantifies how much the images themselves leak site identity, the problem Stern et al. flagged. Second, the two central readers' agreement with each other sets a noise ceiling. Third, central-versus-local agreement per site gives a direct map of which sites read differently.

Gap decomposition. For each held-out center $k$, a model trained leave-one-center-out is scored in four ways:

  • $D^{\text{loc}}_{\text{out}}(k)$: against the local labels of center $k$.
  • $D^{\text{cen}}_{\text{out}}(k)$: against the central labels of center $k$.
  • $D^{\text{loc}}_{\text{in}}$ and $D^{\text{cen}}_{\text{in}}$: the same two scores on a held-out split of the training centers.

The reader-shift share is then

$$\rho(k) = \frac{\big(D^{\text{loc}}_{\text{in}} - D^{\text{loc}}_{\text{out}}(k)\big) - \big(D^{\text{cen}}_{\text{in}} - D^{\text{cen}}_{\text{out}}(k)\big)}{D^{\text{loc}}_{\text{in}} - D^{\text{loc}}_{\text{out}}(k)},$$

where $D$ is Dice. The numerator is how much of the local-label gap disappears when every center is scored with one reading convention. $\rho(k)$ near zero means the gap is image shift. A large $\rho(k)$ means much of the gap was reader shift. Confidence intervals come from bootstrapping cases within each center.

Style-separated training. Site identity is available at training time. We therefore train a shared encoder with one output head per training site on local labels, plus a central head supervised only on the small blinded-labeled subset. The site heads absorb each site's drawing habits. The central head is forced to express the common convention on top of the same features. At deployment to an unseen site, only the central head is used.

This design differs from the MINE approach in a specific way. MINE penalizes site information in the features and so also discards morphology that is genuinely site-correlated, such as a different polyp prevalence mix. Our heads leave the features free and model the disagreement at the output, where the re-read shows it lives.

Baselines

The naive reference is a U-Net trained by plain pooled training on all training centers' local labels and scored against both label sets. This shows what an untreated model does and is the denominator for every improvement.

The strongest generic competitor is an ERM++-style recipe (ERM++): automatic learning-rate scheduling, retraining on the full train-plus-validation data, robust initialization, and parameter averaging. It gets the same backbone and the same hyperparameter search budget as our method. The only reported number is on DomainBed, where it raised ViT-B/16 from 61.1% to 78.6% average accuracy; no reported result found on Lacuna for ERM++ on PolypGen or M&Ms.

The strongest polyp-specific architectures are TransNetR and DPN68×2. TransNetR reports a 2.81% mIoU improvement over the next best model on PolypGen, with no absolute out-of-distribution value in the file. DPN68×2 reports 76.19% Dice on ETIS-LaribPolypDB, 12.07 points above PraNet. We rerun both on our leave-one-center-out splits, keeping their published training settings and giving them the same compute budget as our method. The decomposition is applied to them too, which tells us whether architectural gains come from fitting image shift or from fitting local style.

The strongest invariance competitor is MINE with site as the labeled confounder, following Müller et al., who report 87.7 inverted-split AUROC on confounded CheXpert. No reported result found on Lacuna for MINE on multi-center segmentation. It receives the same site labels our heads receive.

Two ablations isolate the key ingredient. The first uses site heads with no central head, deploying an average of the site heads; this tests whether style modeling alone helps without blinded labels. The second fine-tunes a single head on the central subset; this tests whether the blinded labels help without style separation.

Our method counts as winning if, scored against held-out central labels, it beats ERM++ and MINE by more than the bootstrap interval, and the gain is concentrated at centers with a large $\rho(k)$. A uniform gain across centers would point to ordinary regularization rather than style separation.

Why not just…?

Why not just measure inter-site annotation disagreement and stop there? That measurement is the first product of the re-read and is useful on its own. Label disagreement alone, however, does not say how much of a model's cross-center drop it explains: a site can read differently in regions where models already fail for image reasons. The $\rho(k)$ decomposition is what converts disagreement into a statement about generalization.

Why not just learn per-annotator noise models from the existing labels, without any re-reading? Without a common reference, a site's drawing style and that site's image appearance are confounded. A model cannot tell "this center draws wider borders" from "this center's images blur borders." The site-heads-only ablation is exactly this approach. If it matches the full method, the blinded subset is unnecessary and we will report that.

Why not just use stronger augmentation and a tuned ERM++ recipe? These methods target image statistics. The reader-shift component is, by construction, invisible to them. The evidence is still worth taking seriously: ERM++ beats specialized methods on DomainBed, and AMOS found a classic 3D-UNet ahead of Swin-UNetr on CT. If ERM++ scored on central labels closes most of the gap and $\rho(k)$ is small, then image shift dominates and our method adds nothing. That outcome would still be a useful negative result.

Why not just apply MINE with site as the confounder? This is the real threat, because site labels are available and MINE worked where rebalancing did not in Müller et al.. Its limitation for this problem is that removing site information from the features cannot correct a biased target: a model that is invariant to site still learns the average local convention. The head-to-head comparison on central labels, reported per center, settles which mechanism matters.

Decisive Checks

Three results decide the project. The first is whether the pooled-center bootstrap interval for $\rho$ excludes zero on either benchmark. If it does not, the headline claim fails. The second is whether the readers' site-guessing accuracy is near chance. If it is high, as Stern et al. suggests it might be for strongly shifted modalities, the central labels may carry residual site bias, and we will report $\rho$ separately for cases where readers guessed the site wrong. The third is whether the per-center gains of the style-separated model track $\rho(k)$. That correlation is the evidence that the method fixes reader shift rather than just regularizing.

Across all three checks we report Dice and 95th-percentile Hausdorff distance, both against the ceiling set by agreement between the two central readers. This keeps any claimed improvement from exceeding what human readers themselves can reproduce.