Choosing Pretrained Models for the Population You Will Actually Serve

Clinical teams that build risk models on small cohorts increasingly start from a pretrained network. They pick the network with a cheap score, a "transferability estimate" that rates how well a frozen model's features fit the target task without training it, so they never have to fine-tune dozens of candidates. Those scores are computed on whatever validation patients happen to be at hand, and in small clinical cohorts that sample is usually younger than the population the model will serve once deployed. Epidemiology has a precise name for why this matters. Under the rate-volume divergence, per-person disease rates fall while absolute case counts rise, because the number of older people grows faster than their risk drops. Global ischemic heart disease deaths rose 60.43% from 1990 to 2019 while age-standardized death rates fell 30.8% (Risk assessment of dietary factors in global pattern of ischemic heart disease). If the cheap score is dominated by the middle-aged majority of the validation set, it can recommend a model that is best for yesterday's patients and worse for where the cases will be.

We propose to measure and correct this. The corrected score splits the validation set into age strata. It computes a transferability score within each stratum, pulls the noisy estimates from sparse older strata toward the pooled estimate, and then combines the strata using weights proportional to the projected absolute number of cases in each age group: the projected rate times the projected population. The model is still never fine-tuned during selection. The claim we will test is narrow. Under realistic aging shifts, candidate rankings flip between age strata. Off-the-shelf transferability scores then pick the wrong model. A stratified, shrunk, case-volume-weighted score recovers the deployment-optimal choice at close to the cost of the original score.

Core Claim

Let $a$ index age strata, $s_a$ a per-stratum transferability score for a candidate model, $n_a$ the number of validation patients in stratum $a$, $\bar s$ the pooled score, and $w_a$ the projected share of absolute cases falling in stratum $a$ at deployment time. We score each candidate as

$$ S = \sum_a w_a \, \tilde s_a, \qquad \tilde s_a = \frac{n_a \hat s_a + \kappa \bar s}{n_a + \kappa}, $$

where $\hat s_a$ is the raw within-stratum score and $\kappa$ controls how strongly sparse strata are pulled toward the pooled value. The shrinkage form is the standard empirical-Bayes compromise between a high-variance subgroup estimate and a biased pooled one. The weights mirror the epidemiological identity absolute cases = Σ age-specific rate × age-group population. They make selection target the burden the model will face, not the age mix of the cohort that happened to be recruited.

This combines the three directions without giving them equal weight. Cheap model selection supplies the mechanism. Demographic burden supplies the target of selection. Small-data overfitting is the constraint that makes naive stratification fail, which is why the shrinkage term is needed.

Relation to Prior Work

Two results motivate the proposal. First, cheap scores are fragile when benchmarks stop rewarding sheer model size. How NOT to benchmark your SITE metric shows that ranking candidates by parameter count reaches a weighted Kendall's tau of 0.91 on standard benchmarks versus 0.57 for LogME. On the capacity-balanced Meta-Album benchmark the size ranking falls to 0.31 and the best metric reaches only 0.15. Second, in medicine the size heuristic is actively misleading. Accurate brain age prediction with lightweight deep neural networks finds that a small 3D network, the Simple Fully Convolutional Network (SFCN), beats heavier ones and still beats elastic net with as few as 50 training subjects. Its bias-correction step, which reduced the correlation between brain-age error and chronological age from −0.39 to 0.01, is direct evidence that model errors are structured by age.

Three lines of work supply the methods we reuse. LogME scores a frozen backbone by the Bayesian evidence of a linear head fitted to its features. It is about 3000× cheaper than fine-tuning on vision tasks, but it scores a single pooled distribution. On dataset transferability in medical image classification adds an "adaptability" term, a gradient-magnitude ratio taken from one backward pass. Combined with a feature-suitability term, it reaches an average rank of 1.91 across 11 MedMNIST tasks, against 4.00 for LogME and 3.91 for LEEP. It assumes the validation slice matches deployment and contains no age structure. Small Data, Big Decisions shows that architecture rankings found on small subsets remain stable once softmax temperature is calibrated, giving a 4–7× speedup on ImageNet. That paper constrains our design, because the stability holds only above the interpolation threshold. We therefore include small, underparameterized candidates on purpose and report results for them separately.

On the clinical side, UKB-MDRMF and Ferreira et al. show that UK Biobank supports accurate multi-disease prediction. They chose models by conventional full training. Cross-country learning for infectious disease forecasting shows that pooling regularizes small national datasets, cutting Transformer MAE from 2854 to 2113. Our shrinkage toward the pooled estimate applies the same idea, borrowing across age strata instead of across countries. None of these works asks whether the ranking of candidate models depends on which demographic slice is scored. That question is the gap we address.

Research Program

The testbed is UK Biobank. Its imaging participants span roughly 45–80 years, which gives us old strata to evaluate on even though recruitment skewed younger. Target tasks are incident-disease prediction for several age-graded Phecode groups, such as dementia-related, cardiovascular, and musculoskeletal codes, defined as in UKB-MDRMF. Each task uses two input types: brain MRI, and the tabular demographic and blood features.

The MRI candidate pool contains the pretrained SFCN and 3D-ResNet checkpoints from UKBiobank_deep_pretrain. We add backbones we pretrain ourselves at several widths and depths so that capacity varies deliberately, giving roughly 15–20 candidates in total.

For each task we build small selection cohorts of 50, 200, and 1000 participants. These are sampled with the cohort's natural, younger-skewed age mix, which leaves the 70+ strata sparse. The deployment population is a large held-out set reweighted to projected case volumes. We compute these as stratum population share times stratum incidence under an aging scenario, in the style of the aging-adjusted incidence of Global incidence of prostate cancer. We run a mild scenario and a strong one.

Ground truth comes from fully fine-tuning every candidate on every selection cohort, with five seeds each. This oracle step is paid once to build the benchmark. Selection methods never use it.

We will implement stratified versions of LogME and of the suitability-plus-adaptability score. We set $\kappa$ by leave-one-task-out cross-validation, so it is never tuned on the task being scored. We release the benchmark as a demographic-shift track for transferability estimation.

Baselines

The naive reference point is to pick the largest model, the static size ranking that scored 0.91 on standard benchmarks and 0.31 on Meta-Album (How NOT to benchmark). We also report a second naive point: always picking the lightweight SFCN, since Accurate brain age prediction found it strong. Unmodified LogME, pooled over the selection cohort, is the standard cheap score.

The strongest existing competitor is the suitability-plus-adaptability metric of On dataset transferability in medical image classification. It reports an average rank of 1.91 over 11 MedMNIST tasks against LogME's 4.00, and it is the only metric in the file validated on small medical datasets. We know of no reported result on Lacuna for any transferability metric under a demographic or age shift.

We also run the calibrated small-subset fine-tuning of Small Data, Big Decisions. It sits between cheap scores and brute force in cost, so we report its compute alongside its accuracy.

All methods see the same selection cohort and the same candidate pool. Every method with a free parameter gets the same leave-one-task-out tuning protocol. We report wall-clock GPU time for each method.

Two ablations isolate the key ingredient. The first applies case-volume weights without shrinkage. The second applies shrinkage with uniform weights. A third variant weights each patient individually by the inverse of their age's sampling rate, which is the textbook covariate-shift correction applied directly to the existing score.

Success means two things. First, the stratified score must achieve higher weighted Kendall's tau with the deployment-weighted fine-tuning ranking than the adaptability metric. Second, it must achieve lower top-1 regret, defined as the deployment-weighted AUROC lost relative to the oracle's choice. Both must hold at the 50- and 200-participant cohort sizes, across seeds, at under 2× the cost of the unmodified score.

Why not just…?

Why not just reweight each patient by inverse sampling probability inside LogME? This is the most serious threat. If per-patient weighting matches our score, the contribution shrinks to the empirical finding that rankings flip, plus a recommendation to reweight. We expect it to fall short in the smallest cohorts. With only a handful of 75+ participants, a few large weights dominate the evidence estimate, and the resulting variance is exactly the memorization problem small-data regularization addresses. The third ablation tests this directly, and we will report the outcome whichever way it falls.

Why not just pick the biggest, or the smallest, model? The size ranking's collapse to 0.31 on Meta-Album, and SFCN's advantage over deeper networks, show that neither fixed rule holds in general. Our candidate pool deliberately mixes capacities so that both rules are tested on the same footing.

Why not just fine-tune everything on a small calibrated subset? This approach is accurate above the interpolation threshold. But it still trains every candidate, and its stability guarantee does not cover the underparameterized models that do well in neuroimaging. We will report where it wins on accuracy and how much more compute it uses than our score.

Why not just use age-standardized evaluation? Age standardization deliberately neutralizes demographic change. That is precisely what hides the 60% rise in heart-disease deaths, so it would point selection away from the population where the cases will be.

Decisive Checks

The first check comes before any method comparison. Across tasks and backbones, we compute the Kendall's tau between the fine-tuned ranking in the 45–60 stratum and in the 70+ stratum. If that tau is near 1, rankings do not flip, the problem is not real, and we stop and report that negative result. If rankings do flip, the second check asks whether unweighted scores pick models that lose measurable deployment-weighted AUROC under the strong aging scenario. The third check is whether the stratified, shrunk score closes that gap better than per-patient reweighting does, especially at 50 participants. We will also test whether the same weighting fixes source-country selection on the European forecasting data of Cross-country learning. That test shows whether the idea extends beyond imaging.