Medical imaging models are usually tested for robustness by leave-one-hospital-out evaluation. You train on all hospitals but one, test on the one you left out, and repeat for each hospital. A drop in performance is then read as evidence that scanners, stains, or protocols at the new site confuse the model. But hospitals also differ in how common the disease is. A referral center sees far more severe cases than a community clinic. Leaving out a high-prevalence hospital therefore also changes how common the disease is in the training set. A recent result on ordinary leave-one-out cross-validation shows this kind of coupling between what is held out and the training-set label mix can systematically push predictions away from the true label. On pure noise it can drive the measured area under the ROC curve (auROC) as low as 0.25 (Distributional bias compromises leave-one-out cross-validation).
This note proposes to measure how much of the reported cross-hospital performance drop on imbalanced medical data comes from this evaluation artifact and from ordinary label-prevalence shift, rather than from shifts in image appearance. We also propose a cheap audit that separates the two. The two-sentence version is this. When you hold out a whole hospital, you also change the disease prevalence the model is trained on, and that alone can make the held-out hospital look badly served. We will quantify this effect on standard multi-center benchmarks, show when it matters, and give a prevalence-corrected protocol so that "domain shift" numbers mean what they claim.
Core Claim
Leave-one-center-out (LOCO) evaluation on class-imbalanced, multi-site medical data mixes three separate effects:
- Fold-level distributional leak. Austin et al.'s mechanism, lifted from single samples to whole sites.
- Genuine label-prevalence shift between training sites and the test site.
- Genuine appearance shift, meaning the images of a given class look different at the new site.
Only the third is what domain-generalization methods are built to fix. We claim the first two are separable with a simple logit correction and controlled experiments. We expect them to be large for threshold-based clinical metrics (sensitivity, specificity, accuracy, Dice) and for auROC pooled across folds. We expect them to be small for auROC computed within a single held-out site, which only depends on rankings.
The new contribution is not a training method. It is a site-level evaluation audit that does three things:
- It extends rebalanced cross-validation from single held-out samples to held-out hospitals.
- It replaces Austin et al.'s sample-discarding fix with reweighting, which wastes no scarce positive cases.
- It reports how much of each benchmark's cross-site gap survives once prevalence is accounted for.
Research Program
The prior work plays three roles here.
Motivating work. Distributional bias compromises leave-one-out cross-validation proves the leak exists when one sample is held out. Its fix, Rebalanced LOOCV (RLOOCV), discards an opposite-class training sample each fold, and it raised median auROC from 0.74 to 0.81 across 49 UCI datasets. That paper never considers holding out a whole cluster, and it does not consider appearance shift. The domain-shift literature supplies the other half. WILDS reports Camelyon17 accuracy falling from about 93% in-distribution to about 70% on held-out hospitals. AMOS reports a drop of over 20% mDice on MRI from unseen scanners. Both attribute these drops to site differences without separating out prevalence.
Constraining work. Robust Classification under Class-Dependent Domain Shift shows that when class balance correlates with domain, networks first learn the real signal and then latch onto domain shortcuts. This warns us that a prevalence correction applied only at test time cannot undo shortcuts learned during training. Our audit must therefore also change training-time prevalence, not only adjust outputs.
Method-supplying work. Federated Learning with Classifier Shift for Class Imbalance shows that a per-site logit offset equal to the log ratio of local to global class frequency aligns local and global optima under label skew. It explicitly leaves appearance shift unaddressed. We reuse that offset as a measurement tool rather than a training trick.
For a model trained on sites with class frequencies $\pi_{\text{tr}}(k)$ and tested on a site with class frequencies $\pi_{\text{te}}(k)$, the prior-corrected score for class $k$ is
$$\tilde{f}_k(x) = f_k(x) + \ln\frac{\pi_{\text{te}}(k)}{\pi_{\text{tr}}(k)},$$
where $f_k(x)$ is the model's logit. We compute it with the true test prevalence (an oracle upper bound) and with prevalence estimated from unlabeled test predictions (the deployable version).
We then split the cross-site gap into two parts. The prevalence part is the difference between corrected and raw held-out performance. The residual is the remaining distance to in-distribution performance, which we attribute to appearance shift.
The protocol itself, which we call rebalanced LOCO, has two variants. Both fix the training label mix to the same value in every fold:
- Discarding variant. A direct site-level port of RLOOCV, which removes training samples to restore the label mix.
- Reweighting variant. Assigns per-class loss weights so the effective training prevalence is constant. This avoids throwing away rare positive cases.
Baselines
The naive reference point is standard LOCO evaluation of a model trained by pooled empirical risk minimization at natural prevalence. We report three metrics:
- within-site auROC;
- auROC pooled across folds;
- sensitivity at the threshold chosen on training-site validation data.
For Camelyon17-WILDS, published held-out-hospital accuracies for this setting are 73.3% (Model-Based Domain Generalization) and 75.3% (Out-of-Distribution Robustness via Targeted Augmentations). Our ERM reproduction should land in that range before any correction is credited.
The strongest existing evaluation correction is RLOOCV ported to sites by discarding samples. We run it on identical splits, architecture, and tuning budget, with the same number of seeds and hyperparameter trials as the naive protocol. The reweighting variant beats it if it matches its bias removal on the random-label control below while showing lower variance across seeds in the low-positive regimes. The source paper's reported gain is a median auROC increase from 0.74 to 0.81 on 49 UCI datasets. No reported result was found on Lacuna for any site-level version.
The strongest existing robustness methods are stain-color targeted augmentation, reported at 90.5% on Camelyon17-WILDS (Out-of-Distribution Robustness via Targeted Augmentations), and MBDG, reported at 94.8% (Model-Based Domain Generalization). We do not try to beat them. We ask whether their margin over ERM is unchanged, shrinks, or grows once both are evaluated under prevalence-corrected LOCO. To keep this fair, both get the same training sites, and their published hyperparameters are re-tuned with the same trial budget as ERM.
The key ablation separates training-time from test-time correction. We compare four settings:
- logit offset only;
- reweighted training only;
- both corrections;
- oracle-prevalence versus estimated-prevalence offsets.
This tells us whether the artifact lives in the learned features or only in the output layer.
Why not just…?
Why not just report auROC computed within each held-out site, which ignores prevalence? This is the most serious objection, and it is partly right. A constant prior offset cannot change rankings within one site, so we predict within-site auROC will show little of the leak. The objection fails in three places:
- Clinical deployment uses thresholds, and sensitivity at a fixed threshold is directly moved by the prior.
- Many studies pool predictions across folds, which is exactly Austin et al.'s mechanism.
- Changing training prevalence can change the learned features, as the two-phase finding of Galstyan et al. suggests, and that would alter rankings too.
The controlled-skew experiment below measures within-site auROC directly. If it stays flat while threshold metrics move, we will say plainly that rank metrics are safe and that the problem is confined to thresholds, pooling, and model selection.
Why not just train every fold with a class-balanced loss? Balanced training makes the model's implied prior the same in every fold. That is essentially our reweighting variant, so this alternative is a candidate answer rather than a competitor. The open questions are three:
- How large the artifact is under the natural-prevalence ERM that benchmarks actually report.
- Whether balancing removes it fully once appearance shift is also present.
- Whether balancing trades it for overfitting on rare positives, a risk Tousignant et al. raise for oversampling.
If balanced loss alone removes the artifact, that is our recommended protocol, and the contribution becomes the measurement showing it is needed.
Why not just apply a domain-generalization method and stop worrying about evaluation? WILDS reports that IRM and CORAL failed to beat ERM on 8 of 10 real benchmarks. More fundamentally, these methods target appearance shift, so they cannot fix a measurement confounded by prevalence. If part of a method's reported gain came from a better-calibrated implied prior, prevalence-corrected LOCO will reveal it. That is the method-margin check below.
Proposed Experiments and Decisive Checks
Random-label null control. Following Austin et al., we shuffle labels within each Camelyon17-WILDS hospital so that no signal exists, while keeping per-hospital prevalence unequal. We train ERM under naive LOCO. The prediction is that pooled auROC falls clearly below 0.5 and within-site auROC stays near 0.5. Rebalanced LOCO should restore pooled auROC to 0.5. A failure here would falsify the claim that the leak transfers to the site level.
Controlled prevalence skew. The five Camelyon17-WILDS hospitals have patch-level labels. We subsample tumor patches per hospital to create a range of prevalence spreads, from equal prevalence to a strongly skewed referral-versus-community split, holding image appearance fixed. The main result is a dose-response curve: the fraction of the held-out gap explained by the prevalence part, plotted against the size of the prevalence spread. At equal prevalence it should be near zero. At strong skew it should be substantial for sensitivity and accuracy.
A setting where we predict appearance shift dominates. We run the same split-and-correct procedure on AMOS segmentation. The per-organ voxel frequencies serve as class priors, and the logit offset is applied per voxel class. For the CT-to-unseen-MRI-scanner drop, we predict the prevalence part explains little, because that drop is driven by modality. For small, rare structures such as the adrenal glands and duodenum, where AMOS reports models struggle, we predict a larger prevalence part. A secondary check uses PolypGen's six centers, with polyp area fraction as the imbalance.
Method margin under correction. We re-evaluate ERM, stain-jitter augmentation, and MBDG on Camelyon17-WILDS under both naive and rebalanced LOCO. We report whether each method's margin over ERM changes.
We will report results as decomposition tables per site and per metric, with seed variance, since the number of sites is tiny. We will also report an influence-style sensitivity check that drops each training hospital in turn. This makes visible whether a single high-prevalence site carries the conclusions, in the spirit of Testing Most Influential Sets. We will not claim that paper's formal test, which is derived only for linear regression.
References
- Distributional bias compromises leave-one-out cross-validation
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Robust Classification under Class-Dependent Domain Shift
- Federated Learning with Classifier Shift for Class Imbalance
- Model-Based Domain Generalization
- Out-of-Distribution Robustness via Targeted Augmentations
- Testing Most Influential Sets