When the Holdout Is a Hospital: Measuring Adaptive Overfitting in Medical Out-of-Distribution Benchmarks

Medical imaging models are judged by how well they transfer to hospitals and scanners they never trained on. The benchmarks that measure this, such as the hospital-split pathology dataset Camelyon17-WILDS or the multi-scanner abdominal dataset AMOS, contain many image patches, yet they contain only a handful of hospitals or scanners. Researchers pick architectures, augmentations and adaptation hyperparameters by scoring many variants against the same one or two "out-of-distribution" hospitals, over and over. We argue that this reuse is the same holdout reuse that adaptive data analysis warns about, and that its effective sample size is the number of independent slides and sites, not the number of patches. At that scale the danger is real. Our proposal is to measure how much of the reported progress in medical domain generalization comes from methods being tuned to the specific benchmark hospitals rather than to unseen hospitals in general, and to test whether a reuse-safe evaluation protocol, whose noise is sized to the true unit of independence, removes that inflation.

This also casts a known worry in a new form. The contamination literature already shows that a model can look robust because it saw the test domain's style during pretraining (In Search of Forgotten Domain Generalization). Repeatedly choosing methods by their score on a fixed shifted hospital is another route by which that hospital's style leaks into the final model, this time through human and automated selection rather than training data. If the leak is large, published rankings of medical robustness methods may not survive a change of test hospital. If it is small, the field gains evidence that its current protocol is sound.

Core Claim

The claim is that adaptive overfitting, which Roelofs et al. found negligible on Kaggle, becomes large and measurable in medical out-of-distribution benchmarks. The two conditions under which their meta-analysis did find overfitting both hold here. A Meta-Analysis of Overfitting in Machine Learning studied 120 competitions and found public and private leaderboard accuracy tracking $y=x$, with overfitting under 1% for top teams. It located the exceptions in holdouts with fewer than roughly 100–200 samples and in public/private splits that were differently distributed. It also explicitly left unsupervised use of test data unexamined. Medical benchmarks combine all three conditions: few independent units per site, a deliberate shift between the tuning hospital and the reporting hospital, and test-time adaptation methods that train on the unlabeled test hospital. We predict that selection on a single shifted hospital inflates reported accuracy by several points, that this inflation grows with the number of configurations tried, and that it disappears once scores are computed with noise calibrated to slide- and site-level sensitivity.

The quantity we estimate is the selection gap. Let $\hat h_k$ be the configuration with the best score $A_{\text{pub}}$ among $k$ tried configurations on the tuning ("public") data. The gap is

$$ \Delta(k) = \mathbb{E}\big[A_{\text{pub}}(\hat h_k) - A_{\text{priv}}(\hat h_k)\big] = \underbrace{\mathbb{E}\big[A_{\text{pub}}(\hat h_k) - A_{\text{pub}}'(\hat h_k)\big]}_{\text{adaptive overfitting}} + \underbrace{\mathbb{E}\big[A_{\text{pub}}'(\hat h_k) - A_{\text{priv}}(\hat h_k)\big]}_{\text{site shift}}, $$

where $A_{\text{priv}}$ is accuracy on hospitals never used for selection and $A'_{\text{pub}}$ is accuracy on fresh slides from the same public hospital that were held back from selection. This split separates the two explanations Roelofs et al. could not disentangle on Kaggle. The first term is overfitting to sample noise. The second is shift between hospitals, including shift the selection process has tuned toward. Standard results on the maximum of $k$ noisy estimates predict that the first term scales roughly as $\sqrt{\log k / n_{\text{eff}}}$. If $n_{\text{eff}}$ counts patches, this term is negligible. If it counts slides, it is not. Which count governs the real behavior is an empirical question this project answers.

Relation to Prior Work

Theory in adaptive data analysis supplies the mechanism and the remedy. Generalization in Adaptive Data Analysis and Holdout Reuse shows that reusing a holdout leaks information about its sample noise into later choices. The same paper introduces a reusable holdout, Thresholdout, which answers queries through a differentially private mechanism: a mechanism whose output barely changes when any single record changes. Interactive Fingerprinting Codes and the Hardness of Preventing False Discovery caps the number of adaptive queries answerable on $n$ samples at $O(n^2)$. Tight Bounds for Answering Adaptively Chosen Concentrated Queries studies what happens when samples are not independent. All of this work assumes a single stationary distribution. That is exactly the gap: it says nothing about a holdout that is itself a shifted domain, or about samples that cluster within slides and sites. We reuse Thresholdout's mechanism but change the privacy unit from patch to slide and site. We then ask empirically whether the change matters.

The contamination literature supplies the framing. In Search of Forgotten Domain Generalization trained CLIP on style-purified LAION subsets and found a 60% relative drop in out-of-distribution accuracy once the test style was absent from pretraining. LiveClin found roughly a 10-point drop for GPT-5 on post-cutoff clinical cases. Both treat contamination as a property of training data. We study a third channel, selection, which neither paper examines. We also add a fourth: test-time adaptation, in which the test hospital's unlabeled images and often its labels (for hyperparameter choice) enter the method.

The medical domain-shift work supplies the methods whose gains we audit. On Camelyon17-WILDS, Extending the WILDS Benchmark for Unsupervised Adaptation reports 70.8% out-of-distribution accuracy for standard training (empirical risk minimization, ERM) and 86.7% for Noisy Student self-training. Out-of-Distribution Robustness via Targeted Augmentations reports 90.5% with targeted stain-color jitter, against a 75.3% baseline. These are the numbers whose portability we test. The same WILDS papers also constrain the proposal. They found that invariance methods rarely beat ERM, and WILDS found that CORAL and IRM failed to improve on it in 8 of 10 datasets. A large selection gap would offer one explanation for why method rankings on these benchmarks are unstable.

Two test-time adaptation papers set the adaptive-updating regime. Private and Stable Test-time Adaptation with Differential Privacy applies per-sample clipping and noise to test-time updates. It reports DP-Tent at 62.9% versus Tent at 60.8% on ImageNet-C. Test-Time Adaptation with Source Based Auxiliary Tasks anchors test-time updates with distillation on source data. Both study how the model overfits a test stream. Neither studies how the evaluation overfits the test hospital, and neither studies medical data. That distinction is what separates this proposal from its closest neighbor.

Research Program

The core artifact is a "shadow leaderboard" for Camelyon17-WILDS, with a replication on AMOS. The configuration pool covers ERM; targeted stain augmentation; Noisy Student; CORAL and DANN (two methods that align source and target features); and test-time adaptation (Tent, DP-Tent and a distillation-anchored variant). It also spans backbones with ImageNet or no pretraining and sweeps over learning rate, augmentation strength and adaptation steps, giving several hundred configurations under a fixed compute budget. Camelyon17 has five hospitals. We enumerate role assignments: three training hospitals, one public hospital for selection, one private hospital for reporting. Within the public hospital we split slides into a selection half and a held-back half. This yields the three-way estimate in the equation above for every rotation.

We simulate the research loop in two ways. The first is non-adaptive: all $k$ configurations are fixed in advance and the best is picked. The second is adaptive: a sequential search, such as successive halving or a simple Bayesian optimizer, proposes each new configuration after seeing earlier public scores. The adaptive version is the closer analogue of a community iterating on a benchmark. We plot $\Delta(k)$ and its two components against $k$. We report uncertainty with slide-level and hospital-level bootstrap resampling, never patch-level resampling. The patch-level versus slide-level contrast is itself one of the results.

On AMOS, the units are scanners and patients. AMOS reports drops above 20% mean Dice on MRI from scanners unseen in training. Its 100 MRI scans spread across scanners fall inside Roelofs' danger zone. There we use nnU-Net-style 3D-UNet configurations and rotate scanner roles in the same way.

The intervention is a domain-level reusable holdout. Public-hospital scores are released only through Thresholdout. Its noise scale is set by the sensitivity of the score to one slide, with a second variant set by sensitivity to one hospital when several public hospitals are pooled. Its query budget is set by slide count, not patch count. The rule we propose is short and could be adopted by benchmark maintainers: count independent units at the level at which data were acquired, and budget queries accordingly.

Baselines

The naive reference point is the official WILDS protocol. Configurations are selected on the designated out-of-distribution validation hospital, and the test hospital's accuracy is reported once. We run it on every rotation with the same configuration pool and search budget as every other arm. This arm defines the gap the other arms must shrink.

The strongest existing evaluation practice is leave-one-hospital-out cross-validation with patch-level confidence intervals, followed by selection of the configuration with the best mean across folds. It is the strongest competitor because it already averages over hospitals, which is what a careful reviewer would recommend. We give it the same configurations and compute. It beats our proposal if its selected configurations are at least as accurate on private hospitals and its reported score is equally close to private accuracy. We suspect it will still overestimate, because every fold is reused across all $k$ configurations.

The strongest existing reuse-safe method is Thresholdout from Generalization in Adaptive Data Analysis and Holdout Reuse, applied at the patch level as its i.i.d. analysis implies. The key ablation isolates the proposal's single ingredient by comparing patch-level Thresholdout with slide-level Thresholdout under identical noise mechanism, threshold and budget, changing only the unit of sensitivity.

For the underlying methods, the strongest reported Camelyon17 result in the file is 90.5% average out-of-distribution accuracy for targeted augmentations (Out-of-Distribution Robustness via Targeted Augmentations), followed by Noisy Student at 86.7% (Extending the WILDS Benchmark). We reproduce both. For AMOS, the file reports 89.04% CT mean Dice for 3D-UNet, but no reported result was found on Lacuna for a specific unseen-scanner MRI Dice value, only the over-20% drop. We would treat the protocol as beating the baselines if it met two conditions together. First, the gap between its reported score and private-hospital accuracy must be at most half that of the official protocol. Second, the private accuracy of the configuration it selects must be no lower than that selected by leave-one-hospital-out cross-validation.

Why not just…?

Why not just report leave-one-hospital-out results with error bars? Error bars describe variance for a fixed model. They do not describe the upward bias from choosing the best of $k$ models on the same folds. That bias is the first term in our decomposition, and cross-validation over five hospitals reuses every fold for every choice. This is a real alternative, and the experiments settle it directly. If cross-validation's gap is already small at large $k$, the reusable holdout is unnecessary and we will say so.

Why not just trust Roelofs et al., who found overfitting below 1%? Their evidence comes from holdouts that were large, identically distributed and never used for unsupervised training. Their own exceptions, small holdouts and shifted splits, describe medical benchmarks. The within-hospital held-back slides test their claim at the right unit. If the adaptive-overfitting term is near zero there, their result extends to medicine and the whole gap is shift. That would be a publishable negative result, and it would redirect effort toward collecting more sites.

Why not just collect more hospitals, as live benchmarks do? More sites are the real cure. But LiveClin reports over 1,700 person-hours and more than $42,000 per release even for question-answer data, and labeled whole-slide or 3D segmentation data costs far more. Our $\Delta(k)$ curves estimate how many independent sites and slides a benchmark needs to support a given amount of reuse. That is the quantity needed to justify such collection.

Why not just make test-time adaptation private, as DP-TTA does? DP-TTA bounds what the adapted weights reveal about individual test images. It does not stop a researcher from choosing the adaptation learning rate by looking at labeled test-hospital accuracy. We include DP-Tent in the pool, and we separately measure the gap for test-time methods when their hyperparameters are tuned on the public hospital versus the private one.

Decisive Checks

Three outcomes would decide the question. The first is whether the adaptive-overfitting term, measured on held-back slides from the public hospital, grows with $k$ at the slide-level rate rather than the patch-level rate. The second is whether the ranking of the top methods (targeted augmentation, Noisy Student, ERM, test-time adaptation) stays stable across hospital rotations. We measure this with rank correlation between public and private orderings. The third is whether slide-level Thresholdout shrinks the gap while patch-level Thresholdout does not. A clear positive result on all three would mean published medical robustness gains carry a measurable selection inflation and that a simple change in counting fixes it. A clear negative result would give the field evidence that its current practice is safe.