Automated Discovery of Training-Free Performance Proxies

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 211 papers16 conceptsfrozen 2026-10-04

Neural Architecture Search (NAS) and model optimization (such as quantization or distillation) typically face a massive computational bottleneck: to know if a specific model configuration is good, you usually have to train it. This "train-then-evaluate" cycle is so expensive that it often limits search spaces or requires hundreds of GPU days.

To bypass this, researchers developed "zero-cost proxies"—mathematical heuristics that score a model's potential at initialization using only a single forward or backward pass. However, human-designed proxies like Synflow or Grad_norm are often brittle, failing to generalize across different architectures or tasks. A new research direction, led by researchers such as Peijie Dong and Lujun Li, shifts the burden of proxy design from human intuition to automated symbolic discovery.

The core insight of this field is that the relationship between a model's initial state (its weights, gradients, and activations) and its final trained accuracy is a complex mathematical function that humans are poorly equipped to guess. Instead of manual derivation, these frameworks treat proxy discovery as a symbolic regression problem.

By using evolutionary algorithms, researchers can "evolve" mathematical formulas from a set of primitive operations (e.g., $+, \times, \log, \text{mean}, \text{std}$) and operands (e.g., weight norms, gradient variances, or feature map statistics). As shown in Auto-Prox, these evolved formulas often incorporate non-intuitive mathematical operations that capture structural properties of the network—such as information flow or expressivity—far more accurately than handcrafted metrics.

How Evolutionary Discovery Works

The discovery process generally follows a three-step cycle:

  1. Formula Generation: An evolutionary engine (often genetic programming) creates a population of candidate symbolic expressions.
  2. Correlation Evaluation: Each candidate formula is used to rank a small "benchmark" set of models. The proxy's quality is measured by its correlation (typically Spearman or Kendall Tau) with the models' actual trained performance.
  3. Selection and Mutation: The formulas that best predict performance are selected to "breed" the next generation, undergoing mutations and crossovers to find even more predictive combinations.

This approach allows for the discovery of highly specialized proxies. For example, LPZero by Peijie Dong et al. uses this method to find zero-cost proxies specifically for Large Language Models (LLMs), identifying equations that outperform standard heuristics in ranking Transformer variants.

Generalizing Across Optimization Tasks

The power of automated proxy discovery is its versatility. Because the search process is agnostic to what it is predicting, it can be applied to any optimization task where training is the bottleneck:

  • Mixed-Precision Quantization: In EMQ, Peijie Dong and Lujun Li evolved proxies to predict the accuracy of quantized models, enabling the selection of optimal bit-widths for different layers without any training.
  • Knowledge Distillation: Auto-DAS by Haohui Sun et al. discovers proxies that predict how well a student model will learn from a specific teacher, incorporating teacher-student interaction data directly into the evolved formula.
  • Generative Models: Auto-GAS extends this to Generative Architecture Search, evolving information-aware proxies that slash search times by over 100x compared to training-based methods.

Why Evolved Proxies Work Better

Handcrafted proxies usually rely on a single "signal," such as the gradient norm. In contrast, evolved proxies like those found in Auto-Prox and Auto-DAS tend to be "multi-modal." They might combine the variance of weights in the first layer with the entropy of activations in the last layer, weighted by a logarithmic transform of the total parameter count.

By searching the space of all possible combinations, these algorithms find the specific "fingerprints" of high-performance architectures that are hidden in the model's initial state. This moves the field of AutoML away from expensive trial-and-error and toward a regime where we can "calculate" the potential of a model before a single weight is updated.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 8 papers
  2. Version 1 · 2026-10-04 · 11 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_automated_proxy_discovery_8509e8e7,
  title = {automated proxy discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-proxy-discovery/rdr_8509e8e72c40448098afe62bd9d6bbc0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_discovery_of_training_free_per_8509e8e7_v1,
  title = {Automated Discovery of Training-Free Performance Proxies},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-discovery-of-training-free-performance-proxies/rdr_8509e8e72c40448098afe62bd9d6bbc0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}