Predicting Transfer Performance Without Fine-Tuning

This direction is merged into another direction: computational cost of model selection. Its last version stays readable and citable here.

Version 1 of 121 papers25 conceptsfrozen 2026-10-04

The standard workflow for modern machine learning is to select a pre-trained model and fine-tune it on a target task. However, as the "model zoo" of available architectures and pre-training datasets grows, selecting the optimal starting point becomes a significant computational bottleneck. Traditionally, practitioners had to perform expensive trial-and-error fine-tuning across dozens of candidates to identify the best performer.

A growing body of research focuses on transferability estimation: predicting how well a pre-trained model will perform on a new task using only a single forward pass or minimal computation. The goal is to find a proxy metric that correlates perfectly with final fine-tuned accuracy without actually running the training process.

The Geometry of Feature Alignment

One of the most intuitive ways to measure transferability is to look at the "geometry" of the features a model produces. If a model’s internal representations already naturally group target classes together, it is likely to transfer well.

Michal Pándy and colleagues introduced the Gaussian Bhattacharyya Coefficient (GBC) in Transferability Estimation using Bhattacharyya Class Separability, which treats each class in the feature space as a distribution and measures how well they are separated. Similarly, Zijian Wang and Liang Zheng proposed the Neural Collapse Transferability Index (NCTI) in How Far Pre-trained Models Are from Neural Collapse on the Target Dataset Informs their Transferability. NCTI measures how close a model's features are to "Neural Collapse"—a state where within-class variance is zero and class means are maximally distant. The closer a pre-trained model is to this geometric ideal on the target data, the less work it has to do during fine-tuning.

Information Theory and Gradients

Beyond simple geometry, researchers have looked at the "coding rate" and gradient dynamics to estimate the effort required for adaptation.

  • Coding Rate: In Frustratingly Easy Transferability Estimation, Long-Kai Huang and Yu Rong introduced TransRate. This metric uses information theory to measure how compactly a model can represent the target data. A high TransRate suggests the model has captured the essential structure of the new task.
  • Gradient Expectation: Rather than looking at static features, Huiyan Qi and Lechao Cheng proposed Principal Gradient Expectation (PGE) in Transferability Estimation Based On Principal Gradient Expectation. By analyzing the gradients of a model (even a randomly initialized one), PGE estimates the "optimization landscape" of the transfer task. This approach is often more stable than feature-based metrics because it captures the potential for the model to change during training.

Robustness in Low-Data Regimes

A recurring challenge across these papers is the "low-data regime." Many metrics that work well with thousands of samples fail when only 10 or 50 samples per class are available. This is critical because transfer learning is most valuable precisely when data is scarce.

Recent work has focused on making these estimators robust to sparse data. For instance, Zixuan Hu and Shixiang Tang developed LEAD in LEAD: Exploring Logit Space Evolution for Model Selection. LEAD uses differential equations and the Neural Tangent Kernel (NTK) to analytically predict how the model's outputs (logits) will evolve. Because it relies on a theoretical framework of training dynamics rather than empirical simulation, it remains accurate even with a handful of samples. Similarly, Ashutosh Soni and Ness Shroff showed in BeST - A Novel Source Selection Metric for Transfer Learning that quantization-based similarity can identify top-tier models using as little as 1% of the target data.

Expanding the Scope of Transferability

While much of the early work focused on image classification, the field is expanding into more complex domains:

By moving from "trial-and-error" to "analytical estimation," these researchers are turning model selection from an expensive art into a predictable science.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_predicting_transfer_performance_without__e59a2620,
  title = {Predicting Transfer Performance Without Fine-Tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predicting-transfer-performance-without-fine-tuning/rdr_e59a26208a9a4ce7a2440a46e63db860},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_predicting_transfer_performance_without__e59a2620_v1,
  title = {Predicting Transfer Performance Without Fine-Tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predicting-transfer-performance-without-fine-tuning/rdr_e59a26208a9a4ce7a2440a46e63db860/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}