Predicting Model Transferability through Feature Evolution Dynamics

This direction is merged into another direction: computational cost of model selection, representation dynamics. Its last version stays readable and citable here.

Version 1 of 137 papers52 conceptsfrozen 2026-10-04

As the ecosystem of pre-trained models (PTMs) expands into massive "model zoos," selecting the optimal starting point for a new task has become a significant computational bottleneck. Brute-force fine-tuning every candidate is no longer feasible. Researchers have turned to Model Transferability Estimation (MTE)—the science of predicting a model's downstream performance using lightweight, training-free metrics.

The central challenge in this field is the "adaptability gap." Early metrics treated pre-trained models as static feature extractors, but recent evidence suggests that a model's current state is a poor proxy for its ultimate potential. The most effective recent methods instead model the dynamic evolution of features during the fine-tuning process.

The Failure of Static Snapshots

Traditional transferability metrics, such as LogME (LogME: Practical Assessment of Pre-trained Models for Transfer Learning) by Kaichao You and colleagues, calculate the compatibility between fixed features and target labels. While fast, these "static" approaches often fail to rank models correctly, particularly when dealing with self-supervised learning (SSL) models or significant domain shifts.

As noted by Wenqi Shao and Xiaotong Li, an SSL model might produce "unorganized" initial features that appear low-quality to a static metric. However, these features often possess a high capacity for reorganization. If a metric only looks at the starting line, it misses the model's "velocity"—its ability to rapidly warp its embedding space to separate new classes once training begins.

Simulating the Training Trajectory

To bridge this gap, a new generation of metrics attempts to "peek into the future" by simulating the optimization process without actually performing it.

One intuitive approach is PED (Potential Energy Decline), introduced by Xiaotong Li. By treating data clusters as physical particles that repel each other, PED simulates how features will move toward a stable, class-separated state. Similarly, LEAD (LEAD: Exploring Logit Space Evolution for Model Selection) by Zixuan Hu uses differential equations and the Neural Tangent Kernel (NTK) to predict the evolution of model outputs (logits).

These methods shift the focus from feature quality (how good are the features now?) to feature plasticity (how easily can these features be improved?). By evaluating a "simulated future state," these metrics achieve much higher correlations with actual fine-tuning results.

Measuring Adaptability and Robustness

Beyond simulating trajectories, researchers are developing ways to measure a model's inherent "adaptability."

  • Gradient-based Probing: Chuyan Zhang and others have shown that observing how model weights change after just a single epoch of training provides a far more accurate signal than any static snapshot. This "AC-Corr" metric captures the model's immediate responsiveness to the target task's gradients.
  • Feature Perturbation: Prafful Kumar Khoba suggests "stress-testing" embeddings. If a model's class boundaries collapse under slight feature-space noise, it likely lacks the robustness needed to survive the structural updates of fine-tuning.
  • Intra-class Variance: Huiwen Xu introduced TMI, which focuses on how tightly a model clusters similar examples. The tighter the initial clustering, the less "work" the optimization process has to do to achieve high accuracy.

Theoretical Foundations of Feature Learning

The necessity of these dynamic metrics is supported by deep learning theory. Greg Yang and Edward J Hu demonstrated that standard "infinite-width" neural network theories (like the original NTK) fail to explain transfer learning because they assume features stay static. Their Maximal Update Parametrization ($\mu P$) proves that "feature learning"—the actual evolution of internal representations—is the fundamental mechanism that allows knowledge to transfer.

Furthermore, Michael Jordan and Chi Jin have shown that the diversity of pre-training tasks is what allows a representation to become "vanishably" easy to adapt. This suggests that transferability is not just about matching the source and target domains, but about the model's ability to span a broad "representation space" that can be quickly specialized.

Why It Works: The Geometric Perspective

The success of these dynamic approaches can be understood through the lens of Directional Neural Collapse. Achleshwar Luthra and Tomer Galanti argue that effective pre-trained models organize tasks into orthogonal subspaces.

When we fine-tune, we aren't just training a linear classifier on top of a frozen box; we are triggering a geometric transformation where the model collapses variance along specific task-relevant directions. Dynamic metrics work because they estimate the "energy" or "distance" required to complete this collapse. A model that is "closer" to this collapsed state—or has a clearer path toward it—is the one that will ultimately win the race in the model zoo.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_predicting_model_transferability_through_578ba381,
  title = {Predicting Model Transferability through Feature Evolution Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predicting-model-transferability-through-feature-evolution-dynamics/rdr_578ba381b2424123899123f502e17b46},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_predicting_model_transferability_through_578ba381_v1,
  title = {Predicting Model Transferability through Feature Evolution Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predicting-model-transferability-through-feature-evolution-dynamics/rdr_578ba381b2424123899123f502e17b46/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}