When a medical imaging classifier moves from the hospital where it was built to a new one, its accuracy often collapses. On the Camelyon17 histopathology benchmark, a standard model falls from roughly 93% accuracy on familiar hospitals to roughly 70% on an unseen one. The field reads such drops as engineering failures and responds with better augmentation, better alignment, or better calibration. There is a second possibility that nobody measures. The new hospital's images may simply be harder: slides may be sectioned or scanned in a way that erases the cues separating tumor from normal tissue. In that case no model, however well built, could do much better. The quantity that settles this is the Bayes error, the lowest error any classifier could reach on a task. A literature exists for estimating it. That literature has never been pointed at the gap between hospitals.
We propose to split a deployed model's error at a new hospital into three parts, each with a different remedy. The first is an ambiguity floor that no model can remove. The second is information that the source-trained feature extractor throws away on the new hospital's images, which only retraining can recover. The third is error in the final decision layer, meaning the scoring, threshold, or calibration, which post-hoc fixes can repair without touching the regulator-approved model. Each part is estimated with a few hundred labeled target images. We use prior-invariant Bayes estimators, whose answer does not depend on how common the disease is, because disease prevalence genuinely differs across hospitals. We then test the split on synthetic medical data where every part is known exactly.
Core Claim
Write $e_T$ for the deployed model's balanced error on target hospital $T$. Balanced error averages the error on diseased and healthy cases, so it does not depend on prevalence. Write $L^*_T(\cdot)$ for the lowest balanced error achievable on $T$ from a given input. The decomposition is
$$ e_T = \underbrace{L^*_T(x)}_{\text{ambiguity floor}} + \underbrace{L^*_T(\phi_S(x)) - L^*_T(x)}_{\text{representation loss}} + \underbrace{e_T - L^*_T(\phi_S(x))}_{\text{decision-layer loss}}. $$
Here $x$ is the raw image and $\phi_S$ is the feature extractor trained at the source hospitals. The middle term is never negative, because a deterministic function of the image cannot carry more label information than the image itself. The same bound is the basis of the invariance result of Evaluating State-of-the-Art Classification Models Against Bayes Optimality. That paper shows invertible maps preserve Bayes error, and lossy maps can only raise it.
The claim has three parts. The second and third terms can be estimated with modest target labels. The first term can be bracketed from above and below. The resulting attribution predicts which interventions will recover how much error, and those predictions can be checked.
One honesty constraint shapes the design. Any estimate built from data inside a fixed representation measures the Bayes error of that representation, not of the raw image. So $L^*_T(\phi_S(x))$ can be estimated fairly directly. The raw floor $L^*_T(x)$ can only be approached from above, using the strongest target-trained models, plus a learning-curve extrapolation that removes finite-sample bias. We report the floor as a bracket, never as a point.
Relation to Prior Work
Two lines of work motivate the proposal by exposing the gap. WILDS: A Benchmark of in-the-Wild Distribution Shifts documents the Camelyon17 collapse. Out-of-Distribution Robustness via Targeted Augmentations raises out-of-hospital accuracy from 75.3% to 90.5% with stain color jitter, but cannot say whether the remaining 9.5% is headroom or floor. On chest X-rays, Regulating AI Adaptation: An Analysis of AI Medical Device Updates reports cross-site AUC drops of up to 0.18. It also shows that only 1.4% of approved devices report retraining, so knowing whether a fix needs retraining matters in practice. Explaining medical AI performance disparities across sites with confounder Shapley value analysis finds that patient metadata explains on average 27% of cross-site disparity, and a negligible amount between Stanford (SHC) and Beth Israel (BID). Our floor term offers one candidate explanation for that remainder: the target hospital may be intrinsically more ambiguous.
The closest work on estimating error under shift answers a different question. Characterizing Out-of-Distribution Error via Optimal Transport predicts a given model's target error without labels, including on Camelyon17. It says nothing about what any model could achieve, and it assumes the label distribution is the same at source and target. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift and Improved post-hoc probability calibration for out-of-domain MRI segmentation repair the decision layer by rescaling logits or temperatures. They therefore target only our third term, and they cannot tell when that is the wrong term to fix.
The method comes from the Bayes error estimation literature. The direct soft-label estimator of Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification computes $\mathbb{E}[\min(\eta, 1-\eta)]$, where $\eta = P(Y=1 \mid X)$ is the probability of disease given the input. Two works by Ushio, Ishida and Sugiyama constrain how that estimator can be used:
- Practical estimation of the optimal classification error with soft labels and calibration proves that calibration is necessary but not sufficient, which it calls the "calibration trap". It also proves that isotonic regression, a monotone curve fit, recovers clean soft labels when the scores preserve the true ranking.
- Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators supplies estimators of the best achievable balanced error ($\text{BER}^*$) and best achievable AUC ($\text{AUC}^*$). These do not depend on class priors, which is exactly what comparisons across hospitals with different prevalence require.
A further consequence of that theory drives our design. Isotonic calibration of a model's own scores yields the Bayes error of that score, which is an upper bound on the true floor. So the method is only as good as the representation that produced the scores. The $k$-nearest-neighbor learning-curve extrapolation of Estimating the Bayes Risk from Sample Data provides an independent estimate that uses no model scores. The Cover–Hart inequality relating nearest-neighbor error to Bayes error supplies the lower side of each bracket. Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders already uses the $k$-NN expansion as a clinical descriptor, but within a single cohort and never across sites.
Research Program
Estimating the representation's Bayes error. For each target hospital, we freeze $\phi_S$ and estimate $L^*_T(\phi_S(x))$ from $n$ labeled target images, with $n$ ranging from 100 to 2000. Two estimators run in parallel:
- Isotonic estimator. Fit a flexible probe on the frozen features, calibrate its scores isotonically on held-out labels, and apply the prior-invariant $\text{BER}^*$ and $\text{AUC}^*$ estimators.
- Nearest-neighbor estimator. Fit the asymptotic $k$-NN error expansion to learning curves over subsamples, and extrapolate to infinite sample size.
Agreement between the two, and consistency with the Cover–Hart bounds, is our internal validity check.
Bracketing the raw floor. We train an ensemble of the strongest target-trained models we can build: target-only training, source pretraining followed by target fine-tuning, and stain-jitter models. We then run the same two estimators on their scores and features. The minimum across models, extrapolated along its learning curve, is the upper bracket on $L^*_T(x)$. The lower bracket comes from the nearest-neighbor bound on the best features. Every estimate must pass the calibration-trap test of Ushio et al.: an estimated floor that exceeds some model's measured held-out error signals a broken estimator and is flagged, not reported.
Ground truth. Before trusting any real-data number, we build a synthetic medical testbed with exactly known Bayes error, following Evaluating State-of-the-Art Classification Models Against Bayes Optimality. We train class-conditional normalizing flows on Camelyon17 patches downsampled to 32×32. A normalizing flow is an invertible generator that maps images to a simple Gaussian, and that method's exact numerical integration only works at this resolution. We then create synthetic "hospitals" in three ways:
- Invertible color transforms, mimicking stain shift. These leave the floor exactly unchanged, so all degradation is representation or decision-layer loss.
- Raised flow temperature. Increasing the sampling temperature widens the class distributions and raises the floor by a computable amount.
- Non-invertible blur and noise. These raise the floor by an amount we can bound but not compute exactly.
Varying prevalence across the synthetic hospitals tests the prior invariance. The flow method's reported exact floors, such as about 2.46% on CIFAR-10, show that the construction yields nontrivial ground truth. The 32×32 limit is accepted as the price of exactness.
Baselines
The naive reference point is the standard reading: the entire gap between in-hospital and out-of-hospital error is model failure. On Camelyon17 this attributes all of the roughly 23-point drop reported by WILDS to the model. Our method beats this reference only if it attributes a nonzero floor rise whose bracket excludes zero, and only if that attribution is confirmed on synthetic hospitals where the true rise is known.
The strongest competitor is the oracle gap: train the best possible model directly on target data and treat its error as the ceiling. It is simple and intuitive, and it is what a practitioner would try first. For fairness, it receives the same target label budget, the same architectures, and the same tuning budget as our ensemble; in fact it is a component of our upper bracket. We beat it if our decomposition matches the known synthetic terms more closely, particularly at small $n$, where the oracle is biased by finite samples. We also beat it if it correctly separates representation loss from decision-layer loss, which the oracle gap cannot do by construction.
The direct soft-label estimator applied to the source model's raw target probabilities, following Ishida et al., is the obvious lazy use of Bayes estimation under shift. We expect it to fall into the calibration trap because source probabilities are miscalibrated at the target, and we will report how often its estimates exceed measured errors.
Optimal-transport error estimation, as in the COT paper, is run on the same target images to show that it predicts $e_T$ but not the split. No reported result was found on Lacuna for the exact COT/COTT absolute-error scalar on Camelyon17, so we will measure it ourselves.
Three ablations isolate the key ingredients:
- Replace $\text{BER}^*$ with ordinary Bayes error under deliberately shifted prevalence.
- Remove learning-curve extrapolation.
- Remove isotonic calibration.
For the strongest method on the real benchmark, the reference number is 90.5% out-of-hospital accuracy on Camelyon17 with targeted stain jitter. For chest X-rays it is the up to 0.23 AUC recovered by site fine-tuning. Our floor bracket must be consistent with both, never above what these methods achieve.
Why not just…?
Why not just train an oracle model on the target hospital and call its error the floor? This is the real threat. If across synthetic hospitals the oracle gap tracks the true floor as well as our bracket does, the extra machinery is unjustified for the floor term, and we will say so. We expect it to fall short in two ways. First, at clinically realistic label budgets of a few hundred images, the oracle's error overstates the floor, and learning-curve extrapolation exists to correct exactly that bias. Second, the oracle cannot say whether a locked model's loss lives in its features or in its decision layer, which is the question the regulatory setting described by Wu et al. actually poses.
Why not just refit a linear head on the frozen features and see what it recovers? Refitting the head is part of our procedure, not an alternative to it. A refit head's held-out error estimates the second threshold, $L^*_T(\phi_S(x))$, only up to finite-sample and probe-capacity bias. Without the floor bracket, it cannot tell a small remaining gap caused by lost information from one caused by intrinsic ambiguity. The experiment below tests whether the decomposition's predicted recovery matches what refitting actually recovers.
Why not just use label-free error prediction such as COT, since target labels are expensive? Label-free methods estimate how wrong this model is, not how wrong any model must be. COT additionally assumes the source and target share a label distribution, which fails when prevalence differs across hospitals. We concede that our method needs labels. The label-budget sweep will report the smallest $n$ at which the brackets become informative, so that practitioners can weigh that cost.
Why not just explain disparity with patient metadata? Confounder Shapley analysis leaves most cross-site disparity unexplained, and on SHC–BID nearly all of it. Metadata cannot capture ambiguity in the images themselves. Our floor term is a direct test of whether that residual reflects harder images.
Proposed Experiments and Decisive Checks
The first decisive check is synthetic recovery. On flow-generated hospitals, the estimated floor, representation loss, and decision-layer loss must match the known values within bootstrap confidence intervals. In particular, invertible color shifts must yield a floor change indistinguishable from zero.
The second check is predictive, on the real benchmarks: Camelyon17-WILDS, and the NIH, SHC and BID pneumothorax sites. The decomposition is computed once. Its predictions are then tested against interventions it did not see:
- Post-hoc recalibration and head refitting should recover about the decision-layer term.
- Stain jitter and fine-tuning should recover at most the representation term plus the decision-layer term.
- No intervention should push below the floor bracket.
A decomposition that mispredicts these recoveries on held-out target data is falsified.
The third check answers the open Camelyon17 question directly. Is 90.5% near the target hospital's ceiling? We also ask whether the SHC–BID disparity that metadata cannot explain corresponds to a measurable floor difference between the two hospitals. Both answers are reported as brackets under prior-invariant metrics, with the label budget each answer required.
References
- Evaluating State-of-the-Art Classification Models Against Bayes Optimality
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Out-of-Distribution Robustness via Targeted Augmentations
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- Explaining medical AI performance disparities across sites with confounder Shapley value analysis
- Characterizing Out-of-Distribution Error via Optimal Transport
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification
- Practical estimation of the optimal classification error with soft labels and calibration
- Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators
- Estimating the Bayes Risk from Sample Data
- Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders