Zero Cost Proxies for Neural Architecture Search

446 concepts204 papers13 proposals

Finding the optimal neural network architecture for a given task is traditionally a massive computational burden. Standard Neural Architecture Search (NAS) algorithms operate via a "train-and-evaluate" loop: they sample a candidate architecture, train it over multiple epochs, evaluate its validation accuracy, and use that feedback to propose the next candidate. When multiplied across thousands of candidate models, this process consumes hundreds or thousands of GPU hours.

To bypass this bottleneck, a powerful paradigm shift has emerged: Training-Free Performance Estimation. Instead of training candidate networks to convergence, this approach evaluates models at initialization—using randomly initialized weights and a single forward or backward pass of a tiny data minibatch. The mathematical metrics derived from this initial state are known as Zero-Cost Proxies (ZCPs). As noted by Misha Khodak et al. in NAS-Bench-360, these proxies aim to completely replace the computationally intensive training phase, slashing search times from days to minutes.

Why an Untrained Network Reveals Its Future

At first glance, it seems counterintuitive that a network with completely random weights can tell us anything about its fully trained accuracy. Why does this work?

The core intuition is that an architecture's structural blueprint inherently constrains its capacity to learn and represent information, regardless of the specific weight values. Wuyang Chen et al. formalize this in Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics by breaking down a model's potential into three distinct, theory-grounded pillars: Expressivity, Trainability, and Generalization. Zero-cost proxies work by mathematically probing these pillars at initialization.

Probing Expressivity

Expressivity measures the complexity of the functions a network structure can theoretically map. For networks utilizing rectified linear units (ReLU), the network acts as a continuous piecewise linear function. The more distinct linear regions the network can carve out in the feature space, the higher its expressive capacity.

Xingyi Yang et al. demonstrate in Deep Model Reassembly and Yang Liu et al. in DamoFD that counting these linear regions at initialization serves as a highly reliable proxy for ranking candidate architectures. Another common expressivity proxy is the Jacobian covariance matrix, which tracks how distinct inputs map to distinct activation patterns. As analyzed by Colin White and Arber Zela in How Powerful are Performance Predictors in Neural Architecture Search?, computing the correlation of these Jacobians from a single minibatch provides an immediate signal of model quality without a single step of gradient descent.

Probing Trainability

A network can have massive expressive capacity, but it is useless if its loss landscape is untrainable. Trainability determines how easily gradient descent can navigate that landscape to find a good solution.

To measure this at initialization, researchers look to the Neural Tangent Kernel (NTK), which describes the evolution of neural networks during gradient descent. Specifically, the condition number of the NTK matrix indicates how smoothly gradients flow through the network. A high condition number implies a rugged, ill-conditioned loss landscape where gradients are prone to exploding or vanishing. Thorir Mar Ingolfsson et al. leverage this in Reducing neural architecture search spaces with training-free statistics and computational graph clustering to prune low-quality regions of a search space before optimization begins. Similarly, Rob Geada and A. Stephen McGough combine NTK condition numbers (trainability) with linear region counts (expressivity) in SpiderNet to score and rank architectures without backpropagation.

[Candidate Architecture] 
       │
       ▼ (At Initialization / Random Weights)
 ┌─────────────┴─────────────┐
 │                           │
 ▼                           ▼
[Expressivity Check]        [Trainability Check]
 - Linear Region Count       - NTK Condition Number
 - Jacobian Covariance       - Gradient Signal-to-Noise (ZiCo)
 │                           │
 └─────────────┬─────────────┘
       │
       ▼
[Combined Zero-Cost Proxy Score] ──► (Rank & Select Best Model)

Gradient Signatures and Structural Proxies

Beyond landscape geometry, the behavior of the initial gradients themselves offers a window into final performance. Guihong Li et al. introduce this concept via ZiCo, a proxy based on the inverse coefficient of variation of gradients across data samples. By calculating the ratio of the mean gradient magnitude to its standard deviation using just a couple of input batches, they measure the gradient's signal-to-noise ratio. If an architecture exhibits stable, consistent gradient directions across different samples at initialization, it is highly predictive of strong downstream convergence.

In specific architectural domains, even simpler structural properties can serve as effective proxies. For instance, in autoregressive language modeling, Mojan Javaheripi et al. discover in LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models that total decoder parameter count correlates so strongly with final perplexity that it can act as a standalone zero-cost proxy. For Vision Transformers, Qinqin Zhou et al. introduce a "DSS-indicator" in Training-free Transformer Architecture Search to evaluate synaptic diversity and saliency at initialization, allowing them to find optimal transformer designs in hours rather than weeks.

Automating Proxy Discovery

While early zero-cost proxies were manually derived from deep learning theory, designing them requires rare expert intuition. To scale this approach, researchers have turned the optimization lens onto the proxies themselves: using evolutionary algorithms to automatically discover mathematical scoring formulas.

Yash Akhauri et al. pioneer this in EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring by using genetic programming to evolve mathematical expression trees. These evolved formulas combine basic weight and gradient statistics into highly accurate, interpretable scoring functions.

This automated discovery paradigm has quickly generalized across diverse architectures and tasks:

  • Vision Transformers: Zimian Wei et al. develop Auto-Prox, evolving proxies optimized for the unique attention mechanics of ViTs.
  • Knowledge Distillation: Haohui Sun et al. introduce Auto-DAS, which automatically discovers proxies that predict how well an untrained student network will learn when distilled from a larger teacher model.
  • Generative Models: Lujun Li et al. present Auto-GAS, automating proxy discovery for generative architecture search.

Mitigating Bias via Ensembling

Despite their speed, individual zero-cost proxies are not perfect crystal balls. As systematically evaluated by Xuefei Ning et al. in Evaluating Efficient Performance Estimators of Neural Architectures, individual proxies often suffer from structural biases—such as inherently favoring larger parameter counts or penalizing vital operations like skip connections. For example, a proxy might mistakenly rank a "deep and skinny" network highly simply because of its depth, even if it performs poorly in practice.

To combat this, Hsin-Pai Cheng introduces ZiCo-BC, which applies a bias-correction term to balance architectural depth and width, preventing the search from collapsing into inefficient configurations.

The ultimate solution to proxy bias, however, is ensembling. Because different proxies capture different facets of network behavior (e.g., one measures expressivity while another measures gradient flow), combining them yields a more holistic prediction. Bumsub Ham demonstrates this in AZ-NAS by assembling four complementary proxies—tracking feature diversity, gradient stability, and complexity—into a non-linear ranking system. By synthesizing these diverse signals, ensembled training-free methods achieve ranking accuracies that rival traditional training-based search, completing in milliseconds what used to take days.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 247 papers
  2. Version 1 · 2026-10-04 · 311 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_training_free_performance_estimation_4e2ba8bc,
  title = {training-free performance estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-free-performance-estimation/rdr_4e2ba8bc71e74452b09432fa77cce5d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_training_free_performance_estimation_4e2ba8bc_v2,
  title = {training-free performance estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-free-performance-estimation/rdr_4e2ba8bc71e74452b09432fa77cce5d0/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake