Modeling Feature Evolution for Transferability Estimation

22 concepts15 papers1 proposal

When choosing a pre-trained foundation model for a downstream task, full-network fine-tuning is the gold standard for performance. However, fine-tuning dozens of candidate models to find the best match is computationally prohibitive. To bypass this, early model selection frameworks like LogME (Kaichao You et al.; Elisa Bassignana et al.) and PARC (Daniel Bolya et al.) evaluated models as frozen feature extractors. The intuition was simple: if a model's initial, static representations cleanly separate target classes, it should transfer well.

However, this assumption breaks down during full fine-tuning. As Nan Ding et al. and Zhun Deng et al. point out, updating the entire backbone introduces complex optimization dynamics and representation shifts that static metrics cannot anticipate. A model is not a rigid feature extractor; it is a malleable system whose embedding space reorganizes under gradient descent.

The Static-Dynamic Performance Gap

Evaluating a model solely on its initial feature distribution creates severe blind spots. This discrepancy is particularly pronounced for self-supervised learning (SSL) models, such as masked autoencoders.

An SSL model often yields initial representations that appear unorganized or poorly clustered because it was trained on a pretext task rather than explicit class labels. Yet, as Xiaotong Li et al. and Yaoyan Zheng et al. demonstrate, these features possess an immense latent capacity to reorganize and become highly separable during fine-tuning. Static metrics misinterpret this initial lack of structure as poor quality, completely missing the model's adaptability and architectural depth.

Static Evaluation (Frozen Features):
[Pre-trained Model] ──> [Extract Frozen Features] ──> [Measure Clustering/Separation] ❌ Ignores Adaptation

Dynamic Evaluation (Feature Evolution):
[Pre-trained Model] ──> [Simulate Optimization Trajectory] ──> [Evaluate Future Feature Space]  Optimal Selection

This static-dynamic gap also introduces security risks. Bi-Ru Zhu et al. demonstrated that attackers can easily manipulate pre-trained models to exhibit high initial feature clustering—tricking static selection shortcuts—while engineering them to have a low performance ceiling or hidden backdoors after full fine-tuning.

Simulating the Trajectory of Learning

To bridge this gap, recent methods shift from measuring where features are to predicting where features will go. Instead of treating embeddings as fixed points, they model the trajectory of feature evolution. Researchers have developed several distinct mechanisms to capture these dynamics without the cost of full training:

  • Fisher Space Projection: Wenqi Shao et al. introduced Self-challenging Fisher Discriminant Analysis (SFDA). By projecting features into a Fisher space and using a noise mechanism to simulate hard examples, SFDA estimates the dynamic potential for class separation rather than relying on the initial distribution.
  • Physical Potential Energy: Xiaotong Li et al. formulated Potential Energy Decline (PED). They treated data clusters as physical particles that naturally repel one another to reach a stable state. By simulating this physical optimization process, PED uncovers the latent adaptability of self-supervised features.
  • Implicit Evolution Modeling: Yaoyan Zheng et al. developed Implicit Transferability Modeling (ITM), which uses a divide-and-conquer approach to explicitly simulate how the embedding space transforms during optimization, enabling accurate ranking across highly diverse architectures like ViTs and CNNs.

Pervasiveness of Representation Shifts

The necessity of accounting for representation dynamics spans far beyond standard image classification. Across diverse domains, end-to-end updates fundamentally alter feature geometry:

  • Robotics: Kaylee Burns demonstrated that evaluating visual representations as frozen encoders fails to capture how internal attention mechanisms and emergent properties shift during full-backbone fine-tuning, altering the model's ultimate reliability and robustness.
  • Speech Processing: Shuyue Stella Li et al. observed that unfreezing model parameters in cross-lingual speech models fundamentally alters the balance between acoustic features and language-specific structural information.
  • Reinforcement Learning: Erfan Miahi et al. noted that the optimal representational properties (such as sparsity and complexity reduction) for frozen transfer diverge significantly from those required when representations continue to adapt.

Whether dealing with distribution shifts (Saurabh Garg et al.) or dense downstream tasks like semantic segmentation (Siran Dai et al.), feature drift dictates the final outcome.

Recognizing that post-hoc adaptation can be limited if initial features are fundamentally misaligned, frameworks are emerging to proactively shape these dynamics. For instance, Gustav Wagner Zakarias et al. proposed BiSSL, a bilevel optimization framework that jointly optimizes pretext and downstream objectives during a preparatory stage, ensuring that the representation dynamics are guided toward downstream success from the very start.

Key Foundational Work

Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models (2022) — contributed 3 concept elements to this direction.
Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space (2022) — contributed 2 concept elements to this direction.
Evidence \textgreater Intuition: Transferability Estimation for Encoder Selection (2022) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_representation_dynamics_eb21b1a4,
  title = {representation dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-dynamics/rdr_eb21b1a450294636a9ee8624a3960a07},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_representation_dynamics_eb21b1a4_v1,
  title = {representation dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-dynamics/rdr_eb21b1a450294636a9ee8624a3960a07/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake