Validating Small Single-Center Models by Holding Out Their Hidden Subgroups

A clinical model trained on fewer than a hundred patients from one hospital is usually judged by random cross-validation on that same hospital's data. Those random splits share the hospital's scanners, staining batches and referral patterns, so they reward the model for learning local shortcuts. The internal score is then inflated in exactly the way that external validation later exposes. This proposal tests a simple alternative. Embed the cohort with a frozen pretrained vision model, which tends to group images by acquisition style. Cluster the embeddings. Then validate by holding out whole clusters instead of random patients. The claim is that this "leave-one-hidden-subgroup-out" score ranks candidate training recipes (full fine-tuning, parameter-efficient tuning, augmentation choices, learning rates) in close to the order they would rank on unseen hospitals, and that selecting a recipe by it yields better external performance than selecting by random cross-validation.

The motivation is that the two constraints interact at the moment of model choice. With a small sample, the investigator must pick among many regularization choices. Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity shows that the best choice flips with data size and backbone. With a single retrospective center, the only available signal for that choice is biased toward shortcut-using models. Applications of machine learning to undifferentiated chest pain in the emergency department: A systematic review found 87% of 23 reviewed studies at high risk of bias, largely from single-center retrospective designs without external validation. A cheap, data-internal validation procedure that tracks external behavior would change how such studies choose and report models.

Core Claim

Even a single hospital is not one homogeneous environment. It contains several scanner models, stain batches, and eras of protocol. Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models showed that a pretrained masked autoencoder's embeddings cluster by hidden hardware confounders without metadata. It used those clusters to pick a balanced training subset, and 16 images per class beat the full dataset on LEPset. We reuse its clustering step for a different purpose: building validation folds rather than selecting training samples.

Out-of-Distribution Robustness via Targeted Augmentations sets the limits of this idea. It proves that with one training domain, training cannot separate predictive site features from spurious ones. WILDS: A Benchmark of in-the-Wild Distribution Shifts found that invariance-enforcing training objectives failed to beat plain training in 8 of 10 datasets. We therefore do not try to train invariance from pseudo-environments. We make the weaker claim that within-center subgroup structure is enough to rank recipes by how much they rely on local shortcuts. Ranking is a far easier problem than identifying invariant features.

Research Program

The procedure has three steps. First, embed every training image with a frozen foundation encoder, either a masked autoencoder as in the coreset paper or DINOv2. Second, remove the class-mean direction so that clusters reflect acquisition style rather than diagnosis. Cluster within each class, so that holding out a cluster never removes a label entirely. Third, form folds that hold out one cluster per class at a time. Report both the mean and the worst held-out-cluster score.

The number of clusters matters. Density clustering needs hand-tuned radii, a weakness the coreset paper itself notes. We therefore also test k-means with k chosen by a stability criterion. That choice is fixed in advance and never tuned on external results.

Every candidate recipe is scored three ways: with the cluster-out estimate, with standard alternatives, and with its true performance on held-out hospitals. The recipe pool spans the regime where the choice matters. It includes linear probing, full fine-tuning, LoRA, BitFit and Scale-Shift Features across several learning rates, each with and without generic augmentation (random flips, MixUp). On histopathology it also includes the targeted stain jitter of the targeted-augmentation paper. That gives roughly forty configurations per task.

The analysis asks three questions. Does the cluster-out estimate order recipes the way external performance orders them? Does the recipe it selects generalize better? Does the gap between random-split and cluster-out scores predict how much a model will degrade externally?

Baselines

The naive reference point is random stratified five-fold cross-validation on the single-center cohort, which is what most retrospective studies report. Beating it means the external score of the recipe it selects is lower than ours by more than the seed-to-seed standard deviation, and its rank correlation with external performance is lower.

A stronger classical competitor is bootstrap optimism correction. It resamples the cohort and subtracts the estimated overfitting from the apparent score. This is the standard remedy in clinical prediction modeling. It corrects for small-sample noise but not for shared site structure, and it gets the same candidate pool and compute.

The most dangerous baseline is metadata-grouped cross-validation. It groups by patient, slide, or recorded scanner model wherever such tags exist. On Camelyon17-WILDS, slide identifiers are available within each hospital. If grouping by metadata matches latent clustering, then latent clusters only add value where metadata is missing or confounders are unrecorded. We will report the comparison either way.

A decision-rule baseline skips validation entirely and always picks LoRA for transformers and Scale-Shift Features for convolutional networks. These are the defaults Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity found best in small-data regimes; it reports a 22% improvement for Scale-Shift Features on BreastUS.

Two training-side methods are included for the final external-accuracy comparison. The first is confounder-aware coreset fine-tuning from the coreset paper. The second, on Camelyon17, is targeted stain augmentation. For this benchmark the targeted-augmentation paper reports average out-of-distribution accuracy rising from 75.3% to 90.5%, trained on the full multi-hospital training split rather than on one small hospital. Our small-sample regime is harder, so that number is context rather than a target.

The oracle selects recipes using the true external hospitals. It sets the ceiling.

Two ablations isolate the key ingredient. The first replaces latent clusters with random groups of identical sizes. The second clusters features from a randomly initialized encoder. If either matches our method, the gain comes from making the validation folds smaller, not from confounder structure.

Why not just…?

Why not just use patient- or scanner-grouped cross-validation, which clinicians already know? Where reliable metadata exists, this is the real threat. It may perform as well, and the experiments compare it directly on Camelyon17 slide groupings. The case for latent clusters rests on confounders that are unrecorded or unrecordable, such as stain batch or reconstruction kernel. The coreset paper found these structuring the embedding space without labels. If metadata grouping wins wherever metadata exists, the contribution narrows to a fallback for untagged data, and we would say so.

Why not just always use parameter-efficient fine-tuning, since it already wins with little data? The PEFT study reports a crossover that depends on data fraction and backbone size. Its evaluation was not built around cross-hospital shift. Causal Fine-Tuning under Latent Confounded Shift shows that pretrained models fine-tuned on a confounded dataset still collapse: F1 fell to about 49.33 under flipped spurious correlations. A low-rank update can still encode a local shortcut. The always-PEFT baseline measures directly whether a fixed default leaves external performance on the table.

Why not just fix the training instead, with targeted augmentation or causal fine-tuning? Those methods need either hand-identified shift physics, as in the targeted-augmentation paper, or a choice of which layer carries the spurious signal, as in causal fine-tuning. Both also add hyperparameters that still have to be selected with single-center data. Our procedure is complementary. It is the selector those methods currently lack, and the candidate pool includes targeted augmentation so the two can be combined and compared.

Proposed Experiments

The main testbed is Camelyon17-WILDS. We train on one hospital subsampled to 32, 64 or 128 patches per class, and treat the remaining hospitals as external. WILDS: A Benchmark of in-the-Wild Distribution Shifts reports plain training at about 93% in-distribution and about 70% out-of-distribution accuracy, which confirms the gap is real. Every hospital takes a turn as the single center.

The segmentation arm uses the multi-center fundus and prostate MRI collections from TinyMIG: Transferring Generalization from Vision Foundation Models to Single-Domain Medical Imaging, with Dice as the metric. That paper reports 90.09% Dice on fundus under leave-one-center-out with full source data. We subsample the source to at most 50 images.

A tabular stretch arm uses the MAGGIC cohorts from Boosting Transfer Learning with Survival Data from Heterogeneous Domains. There the clustering runs on covariates and the metric is C-index. That paper reports a C-index of 0.677 for its transfer method against 0.500 for target-only Cox regression on the Euro cohort. This arm tests whether the idea survives without image embeddings.

Metrics are three. The first is Spearman rank correlation between each estimator's scores and external scores across the recipe pool. The second is the external performance of the recipe each estimator selects, reported as regret relative to the oracle. The third is the calibration of predicted degradation. Results are averaged over at least five subsampling seeds per center.

No reported result was found on Lacuna for any validation estimator's rank correlation with external performance on these benchmarks. The baseline numbers therefore come from our own runs.

The decisive checks are three. First, if cluster-out selection does not reduce oracle regret relative to random cross-validation on Camelyon17 across most source hospitals, the idea fails. Second, if random-group folds of matched size do as well, the mechanism is fold size rather than confounders. Third, if metadata-grouped folds match latent folds everywhere metadata exists, the method is useful only for untagged data. We would also measure how often each situation arises in the benchmark data.

A positive result would give single-center small-sample studies a one-line change to their validation protocol, with evidence that it tracks external behavior.