When a pretrained image model is fine-tuned on a new task, its robustness to distribution shift rises and then falls. Partway through training it handles shifted test sets nearly as well as zero-shot models. By convergence that advantage is gone. The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning documents this pattern for BiT models fine-tuned from JFT-300M and ImageNet-21k, and reports that the usual fixes (replay, regularizing weights toward the initialization) do not keep the peak at full accuracy. Separately, a line of data-pruning work shows that the first few epochs of training reveal which examples matter. Those methods have mostly been developed for training from scratch, where early signals are noisy and must be bought with a warm-up "tax."
This note proposes to use the fine-tuning run itself as the measuring instrument. During a short probe run from pretrained weights, we record when each training example gets learned and how deep in the network its correct prediction first appears. Examples that are learned almost immediately from shallow layers look like shortcuts. Examples learned only after the robustness peak look like dataset-specific idiosyncrasies. We hypothesize that both kinds drive the collapse of robustness. If so, a coreset (a small selected training subset) built from examples learned inside the peak window, at depth, should reach near full-data accuracy on the original task while keeping much more of the peak robustness at convergence.
Core Claim
Pretrained weights remove the main obstacle to early-stage scoring, and they create a timing reference that random initialization lacks. The obstacle is that gradients at a random initialization are meaningless; Lowering the Pre-training Tax for Gradient-based Subset Training spends substantial effort building a synthetic warm start for this reason, but only studies training from scratch. The timing reference is the robustness peak, which exists only under pretrained initialization.
Our claim is that when and where an example is learned, measured against that peak, predicts whether training on it helps or harms robustness at convergence. Plain difficulty does not predict this. The claim is falsifiable. If a random subset of the same size retains as much robustness as our coreset, the effect comes from less data and not from which data, and the claim fails.
Relation to Prior Work
The closest work is NUCS. For transfer learning, NUCS allocates per-class budgets by difficulty under the pretrained model and keeps a moderate-difficulty window within each class. It scores from frozen features, static error-norm snapshots and a ridge-regression surrogate, and it evaluates only in-distribution accuracy. We reuse its per-class budgeting as a component. We replace its static score with a score taken from the fine-tuning trajectory, and we target robustness under shift, which NUCS does not measure.
Three trajectory scorers supply the measurement style, though none has been tested from pretrained weights:
- DUAL combines mean early-epoch error with prediction variance.
- CAT uses perplexity change between early checkpoints in machine translation.
- CLD correlates per-sample loss changes with class validation trajectories. It explicitly leaves fine-tuning dynamics to future work, after observing that trajectories diverge in language model fine-tuning.
Four papers motivate and constrain the design:
- Shortcut Learning Through the Lens of Early Training Dynamics shows that spurious features are acquired in the first two epochs in shallow layers. It measures this with prediction depth, the earliest layer at which a nearest-neighbor classifier on that layer's features gets the label right. It is diagnostic only, and we turn its measurement into a selection rule.
- A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias shows that fine-tuning distorts pretrained features and latches onto simple shortcuts. It fixes only the linear-probe head, because perturbing the full fine-tuning is too expensive. Data selection is a cheaper place to intervene.
- TAGCOS reports, for language model instruction tuning, that early fine-tuning gradients are cleaner than converged ones. This supports scoring inside the early window.
- The robustness-evolution paper reports that training on hard examples gives a higher peak than training on easy ones. This is a constraint: our method must beat simple hard-example selection, not only random selection.
Research Program
The method has four steps.
- Probe run. Fine-tune the pretrained backbone on the full target training set for a short run, about the length of the rise to the robustness peak. Save a few checkpoints.
- Per-example measurements. At each checkpoint, record the loss of each example and its prediction depth. Prediction depth uses nearest-neighbor probes on a few intermediate layers, computed on a random reference subsample to keep cost low.
- Classify examples. Assign each example a learning time (the first checkpoint after which it stays correct) and a depth. Mark as shortcut-like the examples learned before the first post-initialization checkpoint at shallow depth. Mark as idiosyncratic the examples still unlearned at the peak checkpoint, or whose loss keeps oscillating. Mark as the core the examples learned inside the window at depth.
- Select and fine-tune. Fill per-class budgets NUCS-style, drawing mostly from the core and adding a small tuned fraction from the other two groups. Then fine-tune from the original pretrained weights to convergence on the selected subset.
Locating the peak needs a held-out shifted set. This is the method's main dependency, and we treat it directly. For each pretraining-source and target pair, the peak epoch is located once on a development shift set and reused. We also test whether a label-free proxy locates the same epoch. The proxy is the epoch at which the fine-tuned model's disagreement with a frozen-feature linear probe, measured on unlabeled target images, starts rising.
As a byproduct, step 1 answers an open question raised by the pre-training tax paper: does scoring from pretrained weights at step zero agree with scoring after a few epochs? We report rank correlations between these two scorings and against a LiPT-style warm start.
Baselines
Full-data fine-tuning to convergence. This is the do-nothing reference. The robustness-evolution paper reports that effective robustness on ImageNetV2 drops to near zero at the end of training. Effective robustness is shifted-set accuracy above what in-distribution accuracy alone predicts.
Full-data early stopping. The same run is stopped at the checkpoint that matches our coreset model's in-distribution accuracy, and separately at the robustness peak. This separates "less training" from "different data."
Random subset. A uniform random subset of identical size gets the same steps and the same hyperparameter search. It is the key control for the hypothesis that any smaller dataset specializes less.
NUCS. This is the strongest existing transfer-learning coreset method, run with its released procedure on the same backbones. NUCS reports matching full-data accuracy with 30% of samples on CIFAR-100 and Food101, and outperforming the next best method by 5.6% at 90% pruning on CIFAR-100-LT. No reported result was found on Lacuna for NUCS under distribution shift; we measure it.
Hard-example coreset. The highest error-norm examples from the probe run, chosen to test the robustness-evolution paper's hard-example finding.
DUAL and CLD. Both are transplanted onto the pretrained probe run. They test whether any trajectory score works, or whether the peak-relative timing and depth are what matter.
Hardness-promoting linear probing then fine-tuning on full data. This is hp-LP+FT from the feature-distortion paper. It reports over 20% higher randomized out-of-distribution accuracy than standard fine-tuning on Dominoes.
Ablations. These isolate the key ingredient:
- prune only the shortcut tail
- prune only the idiosyncratic tail
- drop depth and use timing alone
- replace the peak checkpoint with a fixed epoch count
All methods share data, backbone, total fine-tuning steps, and an equal hyperparameter budget. The probe-run cost is charged to our method.
Beating the baselines means two things at the same pruning rate:
- in-distribution accuracy within one point of NUCS;
- effective robustness at convergence significantly above NUCS, the random subset and the hard-example coreset, across seeds.
Why not just…?
Why not just stop fine-tuning at the robustness peak? At the peak, the model has not reached its final in-distribution accuracy, which is exactly the trade-off the robustness-evolution paper describes. The matched-accuracy early-stopping baseline tests this head on. If early stopping at equal in-distribution accuracy gives equal robustness, our selection adds nothing beyond compute savings, and we would report that.
Why not just train on a random smaller subset? This is the most serious threat, because fewer examples could mean less specialization regardless of which examples. The equal-size random control, trained for identical steps, settles it. The ablations go further: if pruning the idiosyncratic tail helps while pruning an equal random amount does not, the effect depends on content.
Why not just pick the hardest examples, or use NUCS? The robustness-evolution paper says hard examples raise the peak. It does not say they preserve the peak at convergence. NUCS reports that keeping only the hardest examples hurts transfer accuracy. Our hypothesis is that the late, unstable portion of "hard" examples is what erodes robustness, which the idiosyncratic-tail ablation tests.
Why not just use hardness-promoting linear probing? It changes the head only, and it still trains on all the data. We test it both as a competitor and combined with our coreset. If the two are complementary, the gains should add.
Why not cluster static pretrained embeddings? TAGCOS reports gradient-based selection outperforming static semantic embeddings and K-center selection for instruction tuning. That evidence comes from language, so we include embedding K-center selection as an additional baseline rather than assume the result transfers.
Proposed Experiments
ImageNet setting. The primary setting follows the robustness-evolution paper: BiT backbones pretrained on ImageNet-21k, fine-tuned on ImageNet-1k, with effective robustness measured on ImageNetV2. Pruning rates are 30%, 50% and 70%.
Smaller transfer settings. These follow the feature-distortion paper: CIFAR-10 evaluated on CIFAR-10.1, STL10 and the corruption sets, plus Living17 and DomainNet-Sketch for subpopulation and domain shift.
Shortcut test. On Dominoes, the shortcut-like tail should be enriched for examples whose label is predictable from the MNIST half. We check this directly against the known construction.
NUCS settings. CIFAR-100, Food101 and CIFAR-100-LT confirm that in-distribution accuracy is not sacrificed.
Decisive checks. Three outcomes decide the proposal:
- Our coreset beats the equal-size random subset and the matched-accuracy early-stopping run on effective robustness at convergence.
- The idiosyncratic-tail ablation accounts for most of that gain.
- The label-free disagreement proxy locates the peak epoch to within one checkpoint on most pairs.
If the first check fails, the hypothesis is wrong. If only the third fails, the method still works, but it requires a small shifted development set.
References
- The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
- Lowering the Pre-training Tax for Gradient-based Subset Training: A Lightweight Distributed Pre-Training Toolkit
- Non-Uniform Class-Wise Coreset Selection: Characterizing Category Difficulty for Data-Efficient Transfer Learning
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
- Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning
- Coresets from Trajectories: Selecting Data via Correlation of Loss Differences
- Shortcut Learning Through the Lens of Early Training Dynamics
- A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias
- TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data