Researchers routinely pick among pretrained models by linear probing: they freeze each model, train a single linear layer on top of its features, and rank models by the resulting accuracy. That ranking often fails to match what practitioners get when they fine-tune every weight. On semantic segmentation, the rank agreement between probing and fine-tuning is only Kendall's $\tau = 0.38$. On GLUE language tasks, by contrast, probing forecasts fine-tuning well. Nobody can currently say, before paying for fine-tuning, which of these two situations a new task is in.
This note argues that the disagreement has a measurable cause. A linear probe measures how well a model's existing features can be read out. Fine-tuning also rewards how much the model's features can improve on the task. These are two separate quantities, and models differ on the second one. We propose to estimate the second quantity, which we call adaptation headroom, from the frozen model. The estimate is a first-order prediction of how far one fine-tuning step would lower held-out loss, computed with the linear-probe head already in place. We then test whether probe accuracy plus headroom recovers fine-tuning rankings where probe accuracy alone fails. The decisive comparison is against the obvious alternative: a short, real fine-tuning run given the same compute.
Core Claim
The rank flips between probing and fine-tuning are mostly explained by differences in headroom across models, not by noise or poor probe tuning. Headroom can be read off gradients of the frozen model evaluated at the trained probe head. It must be measured at the probe head rather than a fresh one because of how two-stage probe-then-fine-tune training behaves. Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective shows that after probing, the head's weight norm is large (about 24.8 versus 0.96 for a freshly initialized head on the CB dataset). This large head lets small feature changes move predictions a lot, and it preserves pretrained features (cosine similarity 0.804 after probe-then-fine-tune versus 0.336 after plain fine-tuning). Early fine-tuning from a probed head therefore stays close to the pretrained model. A first-order estimate taken at that point should be more faithful than one taken at a random head.
Concretely, let $\theta_0$ be the frozen backbone weights and $V$ the trained probe head. Let $\bar g_{\text{tr}}$ and $\bar g_{\text{va}}$ be the average gradients of the task loss with respect to $\theta_0$ on a training split and a held-out split. To keep this affordable, the gradients are randomly projected to a few thousand dimensions. The predicted fine-tuned held-out loss is
$$\hat L_{\text{FT}} = L_{\text{va}}(\theta_0, V) - \eta \,\langle \bar g_{\text{tr}}, \bar g_{\text{va}} \rangle,$$
where $L_{\text{va}}$ is the probe's held-out loss and $\eta$ is a step length. The correction term is the standard first-order change in held-out loss after a gradient step on the training data. Using separate splits keeps the term from rewarding memorization.
The step length $\eta$ is fixed by one shared rule across all models: a constant step length in parameter space relative to $\|\theta_0\|$. It is not tuned against downstream labels, which avoids the hyperparameter leakage documented in graph representation learning. We also extend the single step to a short trajectory of linearized steps in the projected gradient space, and report both versions.
Research Program
The first piece of work is the estimator. For vision transformers on segmentation, the loss is dense per-pixel cross-entropy, but the gradient is still a single vector per image, so the estimator applies without modification. We compute gradients on 1–4k examples, a sample size motivated by the $1/\sqrt{n}$ convergence that GULP: a prediction-based metric between representations proves for its own estimator. GULP asks the related question of worst-case differences over all linear readouts. It deliberately excludes fine-tuning, and that exclusion is exactly the part we model.
The second piece is an explanatory decomposition. For each pair of models whose order flips between probing and fine-tuning, we ask whether the flip is predicted by their headroom difference. Where it is not, we test two competing explanations. The first is readout pooling. Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification shows that probes on pooled global tokens hide information held in local patch tokens, so some apparent probe failures on dense tasks may be artifacts of how features are read out rather than of feature quality. The second is nonlinearity of the readout. Predicting Fine-Tuning Performance with Probing found that a 20-unit MLP probe forecasts fine-tuning better than a linear one, and that early layers are more predictive than deep ones.
The proposal is shaped by prior work in three ways. Probing's failure on segmentation motivates it: How to Benchmark Vision Foundation Models for Semantic Segmentation? found the gap, and older word-vector evidence (Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance) shows the gap is longstanding. Leakage constrains the design: Overcoming Pitfalls in Graph Contrastive Learning Evaluation shows that tuning choices against downstream labels distorts rankings, so our estimator has no label-tuned knobs. The probe-then-fine-tune analysis supplies the mechanism behind our choice of evaluation point.
Baselines
The naive reference is plain linear-probe ranking with a standard, fixed probe recipe. Kerssies et al. report $\tau = 0.38$ against their fine-tuned reference on segmentation. Any method that fails to clearly exceed this number on the same model pool has not explained anything.
The strongest existing competitor on segmentation is the cheap end-to-end protocol from the same paper. It fine-tunes a ViT-B with a linear decoder and $16\times16$ patches, reaching $\tau = 0.87$ with the expensive ViT-L plus Mask2Former reference while training more than 13 times faster. It is the strongest competitor because it actually fine-tunes. We run it twice. The first run uses its published budget. The second is truncated to exactly the GPU-hours our estimator uses, including the probe training. Our method beats it only if it matches or exceeds the truncated run's $\tau$. Matching the full-budget $0.87$ at a fraction of the cost would be a strong result.
On language tasks, the strongest competitor is the probe-based forecaster of Tamkin et al. It is a regression from three probing task-layer features to fine-tuning accuracy, and it reports an average RMSE reduction of 65.8% (41.7% on RTE, 80.5% on QNLI) against its baseline across 25 model variants. That forecaster is fit across models, while ours needs no cross-model fitting. For a fair comparison we report both RMSE under their leave-one-out protocol and rank correlation. We count it as beaten if we match their RMSE without fitting, or exceed it when our headroom term is added as a fourth feature.
We also include label-free predictors. LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures is included even though it targets probe accuracy rather than fine-tuning accuracy. The file reports its advantage over RankMe only qualitatively; no reported numeric correlation is found on Lacuna. We expect it to track the probe ranking and therefore inherit the probe's mismatch, and confirming that expectation is itself informative.
The key ablation computes headroom at a zero or random head instead of the trained probe head. If the probe-head version is not clearly better, the mechanism we borrowed from the NTK analysis is not doing the work we claim.
Why not just…?
Why not just run a short fine-tune instead? This is the real threat. A truncated real fine-tune measures the actual quantity rather than a linearization of it. Our estimator's advantages are that it has no learning-rate or schedule choices that could leak labels, it is deterministic, and it separates readout from adaptation so that each flip can be explained. None of these matters if a matched-compute short fine-tune ranks models better. The matched-budget comparison above settles the question. If the short fine-tune wins, we report that the decomposition explains the gap but is not needed as a selection tool.
Why not just use MLP or multi-layer probes? Tamkin et al. show these forecast fine-tuning better on GLUE. They still only read out frozen features, however, so they cannot capture a model whose features improve under training. We include a 20-unit MLP probe and probes on concatenated multi-layer features as baselines. We also test whether headroom adds predictive power beyond them. If it adds nothing, headroom is redundant with a richer readout.
Why not just tune the probe harder? Ma et al. show that heavy tuning creates leakage and that some methods remain unstable across hyperparameters regardless. We test this directly: we sweep probe hyperparameters exhaustively and check whether $\tau$ on segmentation rises meaningfully above 0.38. We do not expect it to.
Proposed Experiments
The segmentation regime reuses the 10 vision foundation models, ADE20K, and the fine-tuned reference from Kerssies et al. We also include the GTA V to Cityscapes domain-shift setting, where their cheap protocol reaches $\tau = 0.64$, to test whether headroom behaves differently under distribution shift. The language regime reuses the RoBERTa, ALBERT, and DeBERTa variants and the six GLUE tasks from Tamkin et al. We add BOSS as an out-of-distribution sentiment benchmark, where the NTK paper reports no numbers on Lacuna. The metrics are Kendall's $\tau$ and Spearman correlation against fine-tuned rankings, top-1 selection regret, RMSE where comparable, and GPU-hours.
Three checks decide the question. First, on ADE20K, does probe loss plus headroom exceed 0.38 by a wide margin, and does it match the truncated cheap fine-tune at equal compute? Second, on GLUE, where probing already works, does headroom leave the good ranking intact? That would confirm it corrects failures rather than adding noise. Third, is headroom large and spread out across models on segmentation but small and uniform on GLUE? That pattern would give the first explanation of when probing can be trusted and when it cannot.
References
- Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
- GULP: a prediction-based metric between representations
- Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
- Predicting Fine-Tuning Performance with Probing
- How to Benchmark Vision Foundation Models for Semantic Segmentation?
- Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
- Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks
- LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures