Picking which pre-trained model to fine-tune for a new task is costly, so researchers build "transferability estimators": cheap scores computed from a model's features on the target data that are supposed to predict which model will win after fine-tuning. A recent line of these estimators models representation dynamics, meaning the way a network's internal features reorganize while it is being fine-tuned. They simulate that reorganization instead of scoring frozen features. These estimators are graded against a single "true" ranking, produced by fine-tuning every candidate with one recipe. Benchmarking research in other areas shows that recipe choices such as optimizer, learning rate, augmentation and label smoothing can reorder models by themselves. This proposal asks a question nobody in the transferability literature has measured. Once model size is controlled, how much does the true ranking change when the fine-tuning recipe changes? And do simulated-dynamics estimators beat simply running the real optimizer for a short time at the same compute?
The answer decides how to read the field's headline numbers. How NOT to benchmark your SITE metric showed that on standard suites, ranking models by parameter count reaches a weighted Kendall's τ (a rank correlation that weights the top of the ranking more heavily) of 0.91. On a Meta-Album suite with size-matched architectures, the best metric (SFDA) falls to 0.15. That collapse has two possible readings. Either the estimators are bad, or the target they chase is partly noise once size no longer separates the models. We propose to separate these readings by measuring a recipe noise ceiling, re-scoring dynamic estimators against it, and setting the cost-accuracy frontier that any simulator must beat.
Core Claim
A transferability estimator should be scored against a distribution of rankings induced by a defined family of reasonable fine-tuning recipes, not against one recipe's ranking. The agreement between rankings produced by independent recipes sets an upper bound on what any recipe-agnostic estimator can achieve.
We expect two results on capacity-controlled zoos. First, this ceiling will sit well below 1, so part of the reported collapse reflects an ill-posed target. Second, cheap measurements of real early fine-tuning dynamics, taken under the target optimizer, will track the recipe-marginalized ranking better than closed-form or physics-style simulations of dynamics. Both expectations are hypotheses. The second could fail, and the design is built so that a failure is informative.
Prior Work and Its Role
The motivating evidence comes from two places. The SITE critique above supplies the size-controlled Meta-Album suite, the parameter-count Static Ranker, and the notion of metric fidelity: whether score gaps match accuracy gaps in magnitude. It leaves three things untested: recipe variance in the ground truth, dynamic estimators, and compute-matched real fine-tuning. Revisiting Point Cloud Shape Classification showed that modern training recipes alone lifted PointNet++ on ModelNet40 from 90.0% to 93.3%, enough to overtake newer architectures. It also exposed test-set feedback in model selection. We borrow its discipline: validation-only selection, with the recipe treated as a variable rather than a constant. PUBench (Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms) supplies the template of fixed backbones and fixed tuning budgets, and the warning that validation data must respect the deployment setting.
The estimators under test fall into three groups.
The first group simulates feature movement without running an optimizer.
- PED (Exploring Model Transferability through the Lens of Potential Energy) treats class clusters as elastic spheres pushed apart by spring forces. It raised LogME's τ_w on self-supervised models for CIFAR-100 from −0.008 to 0.667.
- ITM (Implicit Modeling for Transferability Estimation of Vision Foundation Models) replaces gradient descent with a closed-form recurrence toward pseudo class centers. It reports average τ_w 0.61 over ten datasets, against 0.45 for LogME.
Neither method models momentum, AdamW, weight decay or augmentation. Both were validated only on size-confounded suites.
The second group scores geometry directly: SFDA (Not All Models Are Equal) and PEFTDiff (PEFTDiff). PEFTDiff ranks parameter-efficient variants of a single backbone using random-walk distances.
The third group is static and serves as reference: LogME (LogME) and TransRate (Frustratingly Easy Transferability Estimation).
MetaRank constrains our interpretation. It reports H-Score swinging from τ_w ≈ 0.97 on CIFAR-10 to ≈ 0.01 on Aircraft and attributes this to task semantics. Our design tests whether recipe variance explains part of that swing instead.
Research Program
The zoo. The zoo reuses the size-matched transformers from the SITE critique: Twins-SVT, XCiT, CoaT, DeiT, MaxViT and MViT v2. We add self-supervised backbones of comparable size, such as MAE- and DINO-pretrained ViTs, because these are the models whose features reorganize most during fine-tuning.
The tasks. The tasks are the fifteen Meta-Album datasets used in that critique (Meta-Album). We add the Structured group of VTAB-1k (The Visual Task Adaptation Benchmark), where PEFTDiff reports that geometry-based metrics degrade.
The recipe family. The recipe family is a small factorial over:
- optimizer: SGD with momentum vs. AdamW;
- learning rate: a three-point grid;
- label smoothing: on vs. off;
- augmentation: light vs. strong;
- adaptation regime: full fine-tuning vs. LoRA.
Each recipe runs with three seeds. Hyperparameters are always selected on held-out validation data, never on test.
Ground-truth targets. From these runs we build two targets:
- Recipe-marginalized ranking: models ordered by mean test accuracy over the family. This is what a practitioner who has not chosen a recipe faces.
- Best-tuned ranking: models ordered by their best validation-selected configuration. This is what a practitioner with a tuning budget faces.
The recipe noise ceiling. The noise ceiling is the average τ_w between rankings from disjoint recipe subsets, together with the seed-only analogue. The gap between the seed-only and recipe numbers isolates recipe-induced reordering from ordinary training noise.
The new estimator. We build one estimator that uses real dynamics. Each candidate is fine-tuned under the target recipe for a short budget k, measured in hundreds of steps. We then compute LogME and TransRate on the adapted features and combine their change from the frozen state with the early validation loss slope. We call this short-horizon trajectory probing (STP).
The representation-dynamics hypothesis is concrete. How much class separability rises in the first k optimizer steps should predict final rank better than where separability starts (static metrics) or where a simulator says it ends (PED, ITM).
Evaluation metrics. Every estimator is scored on four measures:
- τ_w against each target;
- τ_w divided by the noise ceiling;
- the fidelity measure from the SITE critique;
- top-1 regret, the accuracy lost by deploying the estimator's top pick instead of the true best.
All four are plotted against wall-clock compute on a common GPU.
Baselines
Naive references. The first is a uniformly random ranking. The second is the Static Ranker, which orders models by parameter count. On the Meta-Album suite the SITE critique reports the Static Ranker at τ_w 0.31. Because our zoo is size-matched, we also order models by upstream ImageNet accuracy from the model cards. VTAB's finding that supervised ImageNet models lose to self-supervised ones on structured tasks suggests this proxy will be weak there, but it costs nothing and must be beaten.
Strongest published estimator on this question. The strongest published number on the size-controlled suite is SFDA at τ_w 0.15, reported in How NOT to benchmark your SITE metric. The strongest dynamic estimators by their own reports are ITM (τ_w 0.61 on its standard suite) and PED. There is no reported result on Lacuna for PED or ITM on the Meta-Album size-matched suite, and we will produce one. All three run on the same feature extractions, image resolution and sample counts. Their tunable constants (PED's elasticity and radius scale, TransRate's distortion parameter) receive the same validation-only tuning budget as our method. For a dynamic estimator, beating the Static Ranker on this suite means exceeding τ_w 0.31 against the recipe-marginalized target, with seed-bootstrap intervals that do not overlap.
Strongest unpublished competitor. The most dangerous competitor is plain short fine-tuning: ranking models by validation accuracy after the same k steps STP uses. It is the obvious engineering answer. It is run at exactly matched steps, data and recipe.
Ablations. Two ablations isolate the key ingredient. The first drops early accuracy and keeps only the change in feature geometry. The second keeps only early accuracy. If STP's geometry term adds nothing beyond early accuracy, the representation-dynamics claim fails and we will report that. We will also run every estimator at its native cost and at a compute budget inflated to match STP, so that no method wins merely by spending more.
Why not just…?
Why not just rank by parameter count? On size-confounded suites this heuristic wins outright, at 0.91 against LogME's 0.57. Our zoo is size-matched by construction, and there the same heuristic is reported at 0.31. It remains a reference line rather than a solution. Its residual strength on our zoo will be reported per task group.
Why not just fine-tune briefly and take the early leader? This is a real threat. We do not argue it away. It appears in our design twice: as the strongest baseline and as the "accuracy only" ablation.
Two outcomes would refute the proposal's mechanism:
- early validation accuracy matches STP at every budget, in which case the representation-dynamics component is unnecessary;
- early validation accuracy beats PED and ITM at their own wall-clock cost, in which case simulated dynamics have no practical niche.
Either result is publishable as a corrective. The one place where geometry is expected to help is at very small k on self-supervised backbones. There, early accuracy is dominated by the freshly initialized head, while feature separability is already moving.
Why not just evaluate PED and ITM on the existing Meta-Album suite against its single-recipe ground truth? That would answer whether they collapse, but not why. A low τ_w could mean a bad estimator or a ranking that flips whenever the learning rate changes. Only the noise ceiling separates these explanations. If the ceiling turns out to be near 1, this simpler study would have sufficed, and we will say so.
Why not just let MetaRank pick the best metric per task? MetaRank chooses among static metrics using text descriptions of tasks. If every candidate metric sits near the ceiling-adjusted floor, choosing among them cannot help. We include MetaRank's selections as an extra row, to test whether its gains persist once recipe variance is accounted for.
Decisive Checks
The first check compares recipe and seed variance. If the recipe noise ceiling on the size-matched zoo is not clearly below the seed-only ceiling, recipes do not reorder models there. The "ill-posed target" half of the claim is then false, and the SITE collapse is purely an estimator failure.
The second check is whether dynamics help beyond early accuracy. If STP's geometry-change term does not improve τ_w or regret over the early-accuracy ablation at any k, real-dynamics geometry adds nothing.
The third check is whether simulators have a niche. If short fine-tuning at PED's or ITM's compute cost matches or beats them, simulated dynamics have no remaining advantage on this suite.
The fourth check concerns the dataset swing MetaRank reports. Cross-dataset metric volatility of the kind MetaRank describes should shrink measurably when targets are recipe-marginalized. If it does not shrink, MetaRank's attribution to task semantics stands.
References
- How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation.
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms
- Exploring Model Transferability through the Lens of Potential Energy
- Implicit Modeling for Transferability Estimation of Vision Foundation Models
- Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space
- PEFTDiff: Diffusion-Guided Transferability Estimation for Parameter-Efficient Fine-Tuning
- LogME: Practical Assessment of Pre-trained Models for Transfer Learning
- Frustratingly Easy Transferability Estimation
- MetaRank: Task-Aware Metric Selection for Model Transferability Estimation
- Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification
- The Visual Task Adaptation Benchmark