Predicting Downstream Performance Without Fine Tuning

744 concepts378 papers4 proposals

The modern machine learning landscape is defined by an embarrassment of riches. Repositories host thousands of pre-trained foundation models spanning text, vision, speech, and graph domains. When a practitioner faces a new downstream task, the foundational question is no longer how to train a model from scratch, but rather: which pre-trained model should I use?

Traditionally, answering this requires brute-force fine-tuning—taking a collection of candidate models, training each one on the target dataset, and comparing their final validation scores. As repositories grow, this approach hits a hard computational wall. Fine-tuning dozens of multi-billion parameter models is environmentally unsustainable and financially prohibitive.

To bypass this bottleneck, a rapidly growing body of research focuses on transferability estimation and efficient model selection. The core intuition is that we can predict how well a model will perform on a downstream task by analyzing its internal representations or its statistical alignment with the target data before running a single step of gradient descent.

Evaluating Static Representations

The fastest way to evaluate a pre-trained model is to treat it as a frozen feature extractor. Instead of updating the model's weights, we pass the target dataset through the model once and analyze the resulting embeddings.

Why does this work? If a pre-trained model has already learned a representation space where the target task's concepts are well-separated or geometrically aligned, full fine-tuning will easily converge to a high-performing solution. The geometry of the latent space acts as a proxy for the model's downstream potential.

For instance, You et al. introduced LogME (Logarithm of Maximum Evidence) in LogME: Practical Assessment of Pre-trained Models for Transfer Learning. Instead of training a classifier, LogME computes the Bayesian maximum evidence of a linear model fitted to the extracted features. This approach quantifies how well the frozen features can express the target labels, providing a performance prediction up to 3,000 times faster than full fine-tuning.

This principle has been extended across modalities. For example, Yang et al. successfully adapted LogME and optimal transport metrics to the speech domain in How to Estimate Model Transferability of Pre-Trained Speech Models?, showing that sequential speech features can be ranked in seconds. Similarly, for natural language reasoning, Kadikis et al. demonstrate in Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference that measuring the geometric similarity between sentence embeddings can rank model performance 620 times faster than traditional fine-tuning while maintaining a strong correlation with final results.

Lightweight Probing and Spectral Signatures

If static features are too restrictive, a slightly more expressive proxy is linear probing. Instead of fine-tuning the entire network, we train a single linear layer on top of the frozen backbone. Because linear probing is a convex optimization problem, it is incredibly cheap to compute compared to full backpropagation through hundreds of layers. It acts as a cheap mathematical probe to see how easily the target classes can be separated within the model's hidden states.

In Where to start? Analyzing the potential value of intermediate models, Choshen et al. show that a low-cost linear probing test on a single dataset can reliably rank the quality of intermediate models across various downstream tasks. For structural language tasks like dependency parsing, Müller-Eberstein et al. introduce DEPPROBE in Sort by Structure: Language Model Ranking as Dependency Probing, demonstrating that lightweight linear probes can predict the optimal pre-trained encoder with up to 89% accuracy while using orders of magnitude less compute.

Going a step further, recent theoretical insights reveal that we can sometimes predict fine-tuning dynamics without training any parameters. In Linearization Explains Fine-Tuning in Large Language Models, the authors show that fine-tuning can be modeled via the Neural Tangent Kernel (NTK). At the start of training, the gradients of a neural network behave like a linear system. By analyzing a model's "spectral signature" at initialization, practitioners can mathematically deduce how fast the network will learn the target function and choose which layers to adapt before training even begins.

Meta-Learning and Recommendation Engines

Rather than evaluating every model on the target data from scratch, we can treat model selection as a recommendation problem based on historical performance or metadata. If we know how a set of models performed on past tasks, we can infer their performance on a new task by measuring the similarity between the tasks themselves.

In Guided Recommendation for Model Fine-Tuning, Fowlkes et al. treat model selection as a collaborative filtering task. By training a meta-predictor on historical fine-tuning data and simple features (like model size and dataset difficulty), they bypass the need for exhaustive training.

For vision-language models, Tan et al. propose Model Label Learning (MLL) in Vision-Language Model Selection and Reuse for Downstream Adaptation. MLL pre-characterizes models by their performance on a massive semantic graph of concepts, allowing users to instantly select and ensemble the best models for a specific downstream task without exhaustive testing.

Dynamic Routing and Contextual Bandits

Model selection does not always happen once per dataset; sometimes, the optimal model depends on the specific input query or prompt. In a heterogeneous deployment containing both small, efficient models and large, expensive giants, the goal is to route easy inputs to small models and reserve heavy compute for difficult inputs.

In Capability Instruction Tuning, Zhan et al. introduce MODEL-SAT, which uses pre-computed capability profiles derived from standardized aptitude tests to route user prompts to the best-suited small LLM in a collection, avoiding the linear computational overhead of running every candidate model.

To handle this dynamically in real-time, Hu et al. propose PAK-UCB in PAK-UCB Contextual Bandit: An Online Learning Approach to Prompt-Aware Selection of Generative Models and LLMs. This contextual bandit framework selects the optimal generative model or LLM for each specific prompt on the fly using kernel methods, maximizing output quality while avoiding the prohibitive cost of querying multiple large-scale models for every input.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computational_cost_of_model_selection_9b30df57,
  title = {computational cost of model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-model-selection/rdr_9b30df579ceb495ba275176826c44487},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_computational_cost_of_model_selection_9b30df57_v1,
  title = {computational cost of model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-model-selection/rdr_9b30df579ceb495ba275176826c44487/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake