Racing Short Fine-Tuning Runs to Choose Pretrained Models for Small Medical Datasets

A clinician with 200 labeled scans and a library of pretrained image models has to pick one to fine-tune. There are two standard shortcuts. The first is to fine-tune every candidate, which is expensive. The second is to score each model's frozen features with a cheap formula, which the evidence says does not work on medical data. On skin lesions, one such score, LogME, correlates negatively with the accuracy the models actually reach after fine-tuning (The Performance of Transferability Metrics does not Translate to Medical Tasks). We propose a middle path. Every candidate gets a few hundred steps of real fine-tuning. Each is scored by its calibrated held-out loss, meaning the loss after rescaling its confidence so an overfit model is not penalized for being overconfident. Compute is then spent only on candidates whose scores are still statistically close to the leader.

The bet is that when data is scarce, cheap model selection fails for two reasons. Frozen features cannot see how much a backbone will change when it is adapted to medical textures. And small validation sets produce scores so noisy that most apparent differences between models are not real. A short fine-tune addresses the first problem. Confidence intervals that decide when to stop comparing models address the second. The deliverable is a selector together with a map. The map shows, as a function of labeled sample size, where frozen-feature scores are good enough and where a few steps of adaptation are worth paying for.

Core Claim

Under medical domain shift with fewer than about 100 labels per class, a race of short, calibrated fine-tuning runs selects models with lower regret than frozen-feature transferability scores. Its compute cost stays within one to two orders of magnitude of those scores and far below full fine-tuning. Two ingredients carry the gain: calibrating the loss before comparing candidates, and allocating compute according to how well the noise in the scores lets candidates be told apart. We test each one separately.

The evidence for this claim comes from both research directions. Small Data, Big Decisions trained architectures from scratch on small subsets of natural-image datasets. It found that model rankings stay stable down to about two images per class, but only once cross-entropy is temperature-calibrated. That paper supplies our scoring signal. It did not test pretrained checkpoints, domain shift, or adaptive allocation, and it notes that its result may fail for smaller networks. Rethinking Foundation Models for Medical Image Classification on MedMNIST compared twelve foundation models and found that training a linear classifier on frozen features falls well short of end-to-end fine-tuning on MedMNIST. That finding motivates our turn away from frozen features.

The closest competitor is On dataset transferability in medical image classification. It already adds an "adaptability" term to its score: the ratio of gradient sizes between two layers after one backward pass. Using this, it reaches an average rank of 1.91 across 11 MedMNIST tasks, against 4.00 for LogME. However, it was studied mainly on ResNet-18, and its authors report negative correlations when transferring from natural-image to medical models. We replace its single gradient step with a short adaptation run whose length is chosen per candidate.

A cautionary result also shapes the design. How NOT to benchmark your SITE metric shows that simply ranking models by parameter count scores a weighted Kendall's τ of 0.91 on standard benchmarks, beating LogME at 0.57. This means our evaluation must include diverse architectures, so that "bigger is better" cannot win by default.

Research Program

The selector works as follows. For each candidate, we split the $n$ target labels into $F$ folds. On each fold, we fine-tune end to end for $k$ steps with a fixed, simple schedule. We fit a single temperature on a calibration slice and record the calibrated cross-entropy on the fold's held-out part. Using data from all folds in this way (cross-fitting) keeps every label useful even though labels are scarce.

Let $\hat L_m(k)$ be the mean calibrated loss of model $m$ after $k$ steps, and let $s_m(k)$ be its standard error estimated by bootstrapping over folds and examples. After each round, model $m$ is dropped when

$$\hat L_m(k) - \hat L_{m^*}(k) > z\,\sqrt{s_m(k)^2 + s_{m^*}(k)^2},$$

where $m^*$ is the current leader and $z$ sets the confidence level. The surviving models have $k$ doubled. The race stops when one model remains or when the budget runs out. If several models are statistically tied, it returns the cheapest of them. That fallback is deliberate: How NOT to benchmark your SITE metric documents that many score gaps correspond to accuracy differences that are not significant.

Alongside the selector, we will build an evaluation harness. Its key design choice is to subsample the training set from large labeled datasets while keeping a large test set. This way, the ground-truth ranking reflects what happens when each model is fine-tuned on the same small $n$, and it is measured with little noise. Prior medical studies of transferability scores did not examine how sample size affects them (The Performance of Transferability Metrics does not Translate to Medical Tasks).

We will measure regret against two targets. The first is accuracy on the in-distribution test set. The second is accuracy on a paired external dataset from a different source, because models trained on small data can look excellent on their own test set and collapse elsewhere (Unveiling COVID-19 from Chest X-ray with deep learning).

Baselines

Size ranker (the do-nothing reference). Rank candidates by parameter count, and separately always pick DINOv2. The size ranker is the "do nothing" reference that How NOT to benchmark your SITE metric shows is hard to beat. Always-DINOv2 is what Rethinking Foundation Models on MedMNIST would recommend: it reports 92.02% on DermaMNIST with full fine-tuning, against 88.13% for EfficientNet-B4. Both cost nothing, and the selector must beat both on regret to justify any compute.

Off-the-shelf frozen-feature scores. LogME (LogME) fits a Bayesian linear model on frozen features. TransRate (Frustratingly Easy Transferability Estimation) measures how compressible the features are within versus across classes. We also include LEEP and NCE, the label-based scores that held up best out of distribution in the medical study. All are run on the same $n$ labels. LogME's reported weighted τ on ISIC2019 is −0.067 (The Performance of Transferability Metrics does not Translate to Medical Tasks).

Linear-probe validation accuracy. Train a linear classifier on each model's frozen features, cross-fit on the same folds, and pick the model with the best validation accuracy.

Strongest existing method: NCA plus gradient ratio. This is the combined score from On dataset transferability in medical image classification: leave-one-out nearest-neighbor clustering quality, plus the one-step gradient-ratio adaptability term. Its reported average rank across 11 MedMNIST tasks is 1.91, against 3.91 for LEEP and 4.00 for LogME. It is the strongest competitor because it is the only cheap method that already tries to anticipate adaptation. We will extend its gradient ratio to Vision Transformers using the first two transformer blocks and report both the original and the extended version.

Full fine-tuning of every candidate. Fine-tune each model on the same $n$ with a modest hyperparameter sweep. This gives the upper bound on selection quality and the ceiling on cost.

Fairness and the bar for success. All methods see identical labels and folds. Compute is logged in GPU-seconds, including feature extraction. Each method gets the same small hyperparameter search budget, chosen on a held-out MedMNIST task that is excluded from reporting. Beating the NCA-plus-gradient baseline means lower mean top-1 regret with non-overlapping bootstrap intervals across tasks and seeds, at a budget no more than 50 times larger than that baseline's.

Ablations. Four ablations isolate the key ingredients. The first uses uncalibrated loss. The second uses accuracy instead of loss. The third gives every model the same fixed number of steps instead of racing them. The fourth uses zero steps, which reduces to calibrated frozen features.

Why not just…?

Why not just use plain successive halving on validation accuracy? This is the most serious threat. Our race is structurally a form of successive elimination, and plain halving could perform just as well. We think it will not at small $n$. With 50 validation examples, accuracy has a standard error of several points, so a halving schedule without confidence intervals will drop the true best model early. In addition, Small Data, Big Decisions found that uncalibrated signals are distorted by overconfidence. The accuracy ablation and the uncalibrated ablation test this directly. If either matches the full method, we will report that the contribution is the evaluation map, not the selector.

Why not just pick DINOv2 or the largest model? The benchmark evidence for DINOv2 is real (Rethinking Foundation Models on MedMNIST), but it comes from full training sets. At 20 to 60 images per class, the same paper reports that large models struggle. Separately, ResNet-18 beat ResNet-50 both on small chest X-ray cohorts (Unveiling COVID-19 from Chest X-ray) and at 28×28 resolution (MedMNIST v2). We report always-DINOv2 regret at each $n$. If it is never beaten, that finding is itself the answer.

Why not just use CODA-style active selection? Consensus-Driven Active Model Selection chooses which examples to label next in order to identify the best model. It needs candidates that can already classify the target, which frozen ImageNet or DINOv2 backbones cannot do on a new label set. CODA addresses a different cost, labels rather than compute. It could be layered on top of our short fine-tunes once each candidate has a classifier head. We leave that combination as a follow-up rather than a baseline.

Proposed Experiments

Datasets and sample sizes. The main testbed is MedMNIST v2 (MedMNIST v2) with the 2D tasks at 5, 10, 20, 50 and 100 labels per class. We also use the paired in-distribution and external datasets from the medical transferability study: ISIC2019 with PAD-UFES-20, BreakHis with ICIAR2018, and BrainTumor-Cheng with NINS. Finally, we use the 447-image CORDA chest X-ray set, pretrained on RSNA, with the released pipeline EIDOSLAB/unveiling-covid19-from-cxr.

Candidate pool. The pool includes ResNet-18/50, EfficientNet, several Vision Transformers, DINOv2 and BiomedCLIP, spanning model families so that ranking by size is not trivially correct. Each configuration is repeated over five random subsamples.

Metrics. We report weighted Kendall's τ against the ground-truth ranking, top-1 regret in test accuracy and AUC, and cost in GPU-seconds. Plotting regret against cost gives one curve per method at each $n$.

Four decisive checks. First, at matched budget, the race must have lower regret than NCA plus gradient ratio for $n \le 50$ per class. Second, removing calibration must visibly raise regret. Third, there should be an $n$ above which frozen-feature scores catch up; we will locate that crossover, which is the practical map. Fourth, models chosen on in-distribution calibrated loss must also have low regret on the external dataset. In particular, the race should recover the ResNet-18-over-ResNet-50 reversal on CORDA wherever the ground truth shows it.

Reference numbers. For the frozen-feature baselines, reported numbers exist: τ of −0.067 for LogME on ISIC2019, average ranks of 1.91 versus 4.00 on MedMNIST, and τ of 0.31 and 0.15 for the size ranker and SFDA on Meta-Album. For successive-halving-style selection on medical transfer, there is no reported result on Lacuna, so we will establish that reference ourselves.