Evolutionary Discovery of Zero Cost Performance Proxies

15 concepts8 papers

Deep learning optimization often requires evaluating thousands of candidate neural network configurations. Whether searching for the optimal architecture (Neural Architecture Search), determining which channels to prune, or selecting bit-widths for mixed-precision quantization, the standard evaluation method is prohibitively expensive: one must train the network to convergence to see how well it performs.

To bypass this bottleneck, researchers designed zero-cost (ZC) proxies. These are mathematical heuristics that score a network at initialization using a single forward and backward pass, completely avoiding the training cycle. Historically, proxies like SNIP, SynFlow, or Jacobi covariance were handcrafted by human experts using theoretical insights about gradient flow and weight saliency.

However, human intuition is limited. Handcrafted metrics often fail to capture complex, non-linear interactions within modern architectures like Vision Transformers (ViTs) or Large Language Models (LLMs). To overcome this, a new paradigm has emerged: treating proxy design as a symbolic regression problem and using genetic programming to automatically evolve optimal zero-cost proxies from scratch.


The Mechanics of Symbolic Proxies

Instead of guessing which combination of gradients and weights correlates with final accuracy, this approach treats mathematical formulas as individuals in an evolving population.

To make mathematical formulas searchable, they are represented as symbolic expression trees:

  • Leaf Nodes (Inputs): These are primitive model statistics extracted at initialization. They include weight matrices ($W$), gradient saliency ($G$), activations ($A$), variance, and signal-to-noise ratios.
  • Internal Nodes (Operators): These are basic mathematical operations such as addition, multiplication, logarithmic transforms, absolute values, and information-theoretic reductions (like mean, sum, or variance).
        [ Multiply ]
         /        \
    [ Log ]       [ Sum ]
      /              \
 [ Abs(G) ]        [ Norm(W) ]

By structuring formulas this way, an evolutionary algorithm can navigate a vast search space of computation graphs to discover non-intuitive, branched mathematical formulas that human experts are unlikely to derive. As demonstrated in EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring by Yash Akhauri et al., these evolved formulas capture intricate relationships between gradients and weights that manual heuristic design consistently overlooks.


The Evolutionary Loop

The search engine operates via a classic genetic programming loop optimized for ranking accuracy:

  1. Initialization: A diverse population of random expression trees is generated using a set of mathematical primitives.
  2. Fitness Evaluation: To measure how good a proxy formula is, the algorithm evaluates it on a small, pre-trained benchmark dataset (a "proxy bank"). For every architecture in the benchmark, the formula calculates a score at initialization. The algorithm then computes the correlation—typically using Spearman’s $\rho$ or Kendall’s $\tau$—between the proxy's scores and the actual, fully-trained accuracies. High correlation means high fitness.
  3. Selection and Variation: The best-performing formulas are selected to seed the next generation. They undergo mutation (e.g., swapping an operator or altering a leaf node) and crossover (swapping sub-trees between two successful formulas).
  4. Pruning and Regularization: To prevent the formulas from growing into uninterpretable, bloated expressions, frameworks like LPZero: Language Model Zero-cost Proxy Search from Zero by Peijie Dong et al. introduce rule-based pruning strategies to enforce simplicity and interpretability.

Why Evolved Proxies Generalize

A common concern with evolutionary search is overfitting: will a formula evolved on a small dataset or a specific architecture fail when applied to a new domain?

The evidence suggests the opposite. Because symbolic expressions capture fundamental structural properties of the optimization landscape rather than memorizing data points, they exhibit remarkable transferability. For example, Yuchen Liu et al. in Evolving Transferable Pruning Functions demonstrated that pruning functions evolved using simple statistical metrics on tiny datasets like MNIST successfully transferred to massive benchmarks like ImageNet, achieving state-of-the-art compression tradeoffs on mobile-optimized architectures.

By shifting the search from specific network parameters to the space of mathematical laws, the discovered proxies act as general indicators of a network's capacity to learn.


Specialization Across the Deep Learning Landscape

As neural networks have diversified, the automated discovery of proxies has allowed researchers to break through domain-specific barriers where human heuristics break down.

Hardware-Aware Compression

In mixed-precision quantization, choosing the right bit-width for each layer is a combinatorial nightmare. Handcrafted metrics struggle to predict how quantization noise propagates. In EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization, Peijie Dong et al. showed that evolutionary search could bridge this correlation gap. By combining data sources like gradients and weights in non-intuitive ways, the evolved proxies accurately rank quantized configurations without requiring expensive training-hardware loops.

Standard zero-cost proxies were designed for image classification, but tasks like semantic segmentation or generative modeling have entirely different structural demands.

Transformers and Attention Dynamics

When applied to Vision Transformers and Large Language Models, classic CNN-based proxies fail because they cannot capture the dynamics of self-attention mechanisms. To address this, Zimian Wei et al. developed Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery, finding scoring functions that generalize across diverse vision datasets in just 0.1 GPU days. Similarly, the authors of LPZero used symbolic regression to isolate mathematical relationships uniquely suited to Transformer attention blocks, significantly outperforming human-designed alternatives.

Knowledge Distillation

The paradigm extends even to multi-model interactions. In Auto-DAS: Automated Proxy Discovery for Training-Free Distillation-Aware Architecture Search, Haohui Sun et al. evolved proxies that predict how well a student network will perform when distilled from a larger teacher model. The evolutionary algorithm successfully combined internal student statistics with teacher-student interaction data, creating a highly accurate, training-free predictor for distillation-aware search.

Conclusion

The automated discovery of zero-cost proxies represents a fundamental shift from manual heuristic engineering to algorithmic synthesis. By framing performance estimation as a symbolic evolution problem, this direction enables the rapid, carbon-efficient optimization of neural networks across diverse architectures and hardware constraints.

Key Foundational Work

EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring (2022) — contributed 2 concept elements to this direction.
Evolving Transferable Pruning Functions (2021) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 8 papers
  2. Version 1 · 2026-10-04 · 11 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_automated_proxy_discovery_8509e8e7,
  title = {automated proxy discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-proxy-discovery/rdr_8509e8e72c40448098afe62bd9d6bbc0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_automated_proxy_discovery_8509e8e7_v2,
  title = {automated proxy discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-proxy-discovery/rdr_8509e8e72c40448098afe62bd9d6bbc0/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake