Tune Only When the Evidence Says So: Anchored, Gated Hyperparameter Selection for Datasets Under 100 Samples

When a clinical dataset has 60 patients, standard practice still says to hold some out and pick the hyperparameters (learning rate, weight decay, augmentation strength, fine-tune versus freeze) that score best on them. With a dozen held-out cases, however, one misclassified image moves accuracy by several points. Picking the best of many configurations on such a slice mostly picks the luckiest one. The result looks tuned but generalizes worse than no tuning at all. This proposal treats selection at tiny sample sizes as a hypothesis test rather than an argmax. We start from an "anchor" configuration transferred from related datasets, never from the target data. We then move away from it only when a paired, resampled comparison on the target data shows, with a confidence bound that accounts for how many alternatives were tried, that the alternative really is better.

The second contribution is measurement. We build a controlled benchmark by subsampling large medical imaging datasets down to 20–100 training examples while keeping their large official test sets as ground truth. This lets us measure directly how much tuning helps or hurts in this regime. Evidence for the harm exists only in NLP prompt selection and in tabular data above 500 samples.

Core Claim

Below roughly 100 labeled examples, the useful question is not "which configuration scores best on held-out data" but "is there enough evidence to leave a sensible prior choice." The reason is simple. If $K$ configurations are compared on $m$ held-out examples, and each estimate has noise of standard deviation roughly $\sigma/\sqrt{m}$, the best-looking one is inflated by about

$$\Delta_{\text{noise}} \approx \frac{\sigma}{\sqrt{m}}\sqrt{2\log K},$$

which is the expected maximum of $K$ roughly independent Gaussian errors. Selection therefore helps only when the true performance spread among candidates exceeds $\Delta_{\text{noise}}$. At $m=15$ and $K=50$ that threshold is large.

We claim two things. First, gating selection on a paired lower confidence bound against a transferred anchor gives higher expected test performance than argmax selection under cross-validation. Second, it has a much lower probability of ending up worse than the anchor. The pairing is what makes this feasible at tiny $m$: the candidate and the anchor are scored on the same examples and folds, so shared noise cancels in their difference.

Where This Sits

Several papers motivate the problem. True Few-Shot Learning with Language Models restricted prompt selection to the few-shot training set. There, leave-one-out cross-validation and description-length criteria came 5–7% behind oracle selection on LAMA, carried about a 27% risk of choosing a prompt 13% worse than average, and cost 2–10% on SuperGLUE with ADAPET. That work diagnoses the failure for discrete prompts in NLP but offers no remedy. We supply one and test it in vision, where the hyperparameters are continuous.

Scheffer and Herbrich model tuning as a channel that leaks held-out label information into the final configuration. Their view motivates charging each tried configuration against a fixed error budget, which is exactly what our multiplicity correction does. Rethinking the Setting of Semi-supervised Learning on Graphs shows how such leakage masquerades as architectural progress on Cora. The mechanism is the same as ours, but the setting there is graphs with a validation set larger than the training set.

Other work supplies methods we reuse or test against. Reshuffling Resampling Splits shows that drawing a fresh train/validation split for every configuration keeps black-box optimizers from exploiting fixed-split noise. Its evidence stops at $n \ge 500$ tabular data and explicitly leaves the extreme small-sample regime open. We use reshuffling across repeats, but we deliberately keep the anchor and the candidate paired within each repeat. This resolves a real tension: decorrelated noise protects the search from exploiting a split, while correlated noise is what makes a pairwise test sharp. Pointwise Sampling Uncertainties on the Precision-Recall Curve shows that likelihood-ratio confidence regions stay calibrated down to about 10 records, whereas normal approximations fail below about $10^4$. That paper only reports uncertainty passively. We put interval estimates of this kind inside the selection rule itself.

Some papers constrain the design. Bao et al. bound the generalization gap of gradient-based tuning as growing with outer iterations and shrinking only as $1/m$. Maclaurin et al. and Lorraine et al. show that high-dimensional continuous tuning memorizes validation labels. Together they tell us not to run gradient-based tuning over many hyperparameters at $m<50$, so we keep the candidate set small and discrete.

Wu and Papiez benchmark foundation models on MedMNIST v2. They find that DINOv2 with end-to-end fine-tuning beats linear probing, and they flag 20–60 examples per class as the regime where adaptation becomes hard. That sets both our backbone and our target regime.

Two validation-free alternatives also constrain the design. DARTS without a Validation Set optimizes a training-speed proxy for marginal likelihood but is biased toward deep cells. Last Layer Marginal Likelihood for Invariance Learning learns invariances without a validation split in low data, but fails on CIFAR-10 and does not cover general hyperparameters. We include the first as a competitor rather than build on it.

Research Program

We will build three things.

The first is the benchmark. We take 2D MedMNIST datasets through the official loaders (MedMNIST v2): DermaMNIST, PneumoniaMNIST, BreastMNIST, RetinaMNIST, OrganAMNIST, OCTMNIST and PathMNIST. From each we draw many independent stratified training pools of $N \in \{20, 50, 100\}$. The official test set is split once into two halves. The "oracle half" is used only to compute the best achievable configuration for regret. The "evaluation half" is used only to report final numbers. Because pools are cheap to redraw, we can estimate not just mean performance but the full distribution of outcomes, including the probability of harm.

The second is the anchor. For a target dataset, the anchor is the configuration with the best mean rank across the other MedMNIST datasets at the same $N$ (leave-one-dataset-out), so no target labels are used. The same procedure produces a shortlist of the next $K-1$ most plausible configurations, which keeps $K$ small without looking at the target.

The third is the gated selector. On the target pool we run $R$ reshuffled repeats of stratified $k$-fold cross-validation. Within every fold, the anchor and each candidate are trained and scored on identical examples. For each candidate $c$ we compute the mean per-example loss difference $\bar d_c$ relative to the anchor across repeats. We then compute a lower confidence bound on $\bar d_c$ at level $\alpha/K$, using a variance inflation that accounts for overlap between resampled training sets. The inflation factor is calibrated by simulation on the benchmark itself, so coverage is checked rather than assumed. We adopt $c$ only if its bound is above zero; among passing candidates we take the largest bound. Finally we retrain on the full pool, following the retrain-on-all-labels protocol of Ding et al..

The search space covers learning rate, weight decay, number of epochs, augmentation strength, and linear probe versus end-to-end fine-tuning for a DINOv2 backbone, plus a ResNet-18 control. A deliberately high-dimensional arm, with per-block learning rates (about two dozen continuous values) tuned by random search, tests whether overtuning grows with search dimension at this scale, as the dimensionality literature predicts.

Baselines

The naive reference point is the generic default: a standard AdamW recipe with end-to-end fine-tuning, no tuning, the same backbone and epochs, retrained on the full pool. Any method that loses to this on average is actively harmful, and Perez et al.'s results suggest argmax selection sometimes will.

The strongest non-tuning competitor is the transferred anchor alone, the leave-one-dataset-out configuration used without any target-data selection. This is the ablation that isolates the gate. If our method does not beat the anchor alone, the gain comes entirely from transfer and the gating is unnecessary. For this comparison and for our method, no reported result on Lacuna exists for MedMNIST at $N \le 100$. The only reference number in the file is the full-data regime, where Wu and Papiez report 92.02% accuracy for DINOv2 on DermaMNIST. That is an upper reference, not a small-sample baseline.

The strongest existing tuning method is reshuffled-holdout random search from Schalk et al., with argmax selection. We also run repeated 5-fold cross-validation with argmax selection, which stands in for the nested procedure recommended by Scheffer and Herbrich. Both get the same candidate budget $K$, the same total training runs as our method (we match $R\times k$ fits), and the same retraining step. For the main comparison they also get the same transferred shortlist, so they differ from us only in the decision rule. We additionally report both with an unrestricted random search space, since that is how they are used in practice.

The file reports no numbers for reshuffled holdout on specific datasets, only that it "frequently matched or exceeded" 5-fold cross-validation at a fifth of the cost. No reported result found on Lacuna for either method below $n=500$.

The validation-free competitor is the training-speed estimate of Fil et al.. Here we sum training losses over the run and select the configuration with the lowest sum, using all $N$ examples for training.

Further ablations remove one ingredient at a time. Gating against the generic default instead of the transferred anchor tests the value of transfer. Gating with unpaired estimates tests whether pairing is what makes the gate usable. Gating with fixed rather than reshuffled splits tests Schalk et al.'s mechanism at $N<100$. Dropping the $\alpha/K$ correction tests whether the leakage-budget view matters.

Beating a baseline means a higher mean evaluation-half accuracy (or AUC for binary tasks) across pools and datasets under a paired test across pools, together with a harm probability no higher than that baseline's. The harm probability is the fraction of pools where the chosen configuration scores below the anchor by more than one point.

Why not just…?

Why not just use nested or repeated cross-validation and take the best configuration? At $N=50$, inner folds hold 8–10 examples. The noise formula above says the winner's advantage is then mostly selection noise, and Perez et al. observed exactly this: near-chance selection with a 27% risk of a badly below-average choice. Our experiments measure this directly as regret versus $K$ under argmax cross-validation. If that curve turns out flat on images, the premise is wrong and we will say so.

Why not just skip tuning and use transferred defaults? This is the real threat, and the proposal is built so that it cannot be argued away: the transferred anchor is both our starting point and a headline baseline. If the gate rarely fires, or fires without measurable gain, the honest conclusion is that below about 100 examples one should transfer hyperparameters rather than tune them. That would still be a useful, benchmarked recommendation. What we expect instead is that the gate fires on datasets whose optimum differs from the family consensus, for example the choice between linear probing and fine-tuning on the smallest pools, and does nothing elsewhere. The per-dataset breakdown settles which picture holds.

Why not just reshuffle splits, as Schalk et al. did? Reshuffling removes the optimizer's ability to exploit one fixed split. It does not shrink the per-estimate variance that dominates at $m\approx 10$. The file's own analysis also notes that reshuffles of a tiny pool overlap heavily. We test fixed versus reshuffled splits inside our gate and as a stand-alone baseline. If reshuffled argmax matches our method at $N=100$ but not at $N=20$, that locates where each tool applies.

Why not just avoid held-out data and use a marginal-likelihood proxy? Training-speed selection is biased toward configurations that fit fast. Fil et al. report that bias for depth, and in small medical data fast fitting is often memorization. Tartaglione et al. found that deeper ResNet-50 overfit relative to ResNet-18 on 447 X-rays. The last-layer approach of Schwöbel et al. does not cover learning rates or schedules. We run the training-speed proxy on identical candidates. If it matches our method, it would be preferable because it needs one run per candidate, and we would report that.

Proposed Experiments

The first check establishes the phenomenon. On MedMNIST pools at $N\in\{20,50,100\}$, we plot evaluation-half accuracy of argmax-cross-validation selection against $K\in\{1,5,20,100\}$ and against search dimension. The prediction is that accuracy peaks and then falls, with the peak at smaller $K$ as $N$ shrinks. If accuracy rises monotonically, tuning is safe in this regime and the proposal's motivation fails.

The second check is calibration. Over thousands of simulated candidate–anchor pairs, we compare the gate's lower confidence bounds to the true differences measured on the oracle half. Coverage must be at least $1-\alpha$. Otherwise the variance inflation is recalibrated before any comparison is reported.

The third check is the main comparison. Across seven MedMNIST datasets, three pool sizes and at least 30 pools each, we compare the gated selector with every baseline and ablation above on mean accuracy or AUC, regret to the oracle-half best, and harm probability. Success means beating both the anchor alone and argmax selection on mean performance while keeping harm probability near $\alpha$.

The fourth check tests transfer beyond MedMNIST. First, we select on a single-hospital pool of $N\le 100$ from Camelyon17-WILDS (WILDS) and evaluate on held-out hospitals. The question is whether the gate also avoids configurations that exploit hospital-specific staining; the file's full-data reference points are a baseline of 75.3% and 90.5% with targeted stain jitter in Targeted Augmentations. Second, we run the OpenML tabular suite of Schalk et al. subsampled to $N<100$, directly extending their study below the sample sizes they tested.