Low Fidelity Proxies for Efficient Neural Architecture Search

34 concepts25 papers1 proposal

The primary bottleneck in automated machine learning—whether searching for the ideal architecture, pruning a model, or tuning hyperparameters—is the prohibitive cost of evaluation. Training a single deep neural network to convergence can take days; training thousands of candidates to find the best one is often computationally impossible. To solve this, researchers use low-fidelity proxies: cheap, noisy estimates of a model's final performance that allow search algorithms to navigate vast design spaces without exhaustive training.

The Relative Ranking Hypothesis

The core intuition behind low-fidelity proxies is that we do not need to know a model's absolute final accuracy to determine if it is worth keeping. We only need a reliable relative ranking. If Model A outperforms Model B after just five epochs, there is a high probability it will still outperform Model B after one hundred.

Fei Ye argues that these proxy fitness scores allow search algorithms to discard poor configurations early, minimizing overhead. This strategy is effective because the performance signal often stabilizes long before convergence. For example, Marchisio et al. demonstrated that training capsule networks for only 5 to 10 epochs yields a correlation of over 0.93 with their final fully-trained accuracy.

Dimensions of Fidelity Reduction

Researchers have developed several ways to "lower the fidelity" of an evaluation to save time:

  • Truncated Training: The most common approach is simply stopping early. Wu et al. found that a very low number of epochs is sufficient to distinguish high-quality architectures from poor ones in side-channel analysis. Similarly, Chen et al. and the authors of the AutoG framework use early-stopping metrics to rapidly select graph neural network schemas.
  • Data Subsampling: Instead of training on the full dataset, models can be evaluated on a representative fraction. Wang et al. showed that training on just 10% of the data for 5 epochs provides a sufficiently strong signal for evolutionary search.
  • Model and Task Simplification: Search can be performed on a "proxy task." Ghiasi et al. achieved efficiency by evaluating candidate feature pyramid architectures using a significantly smaller backbone network and reduced training durations.
  • Algorithmic Approximations: In specialized domains like diffusion models, Song et al. found that using a reduced number of denoising steps during lookahead simulations provides a sufficient signal for path estimation in reasoning tasks.

Why a Little Training Beats None

A tempting alternative to low-fidelity training is "zero-cost" proxies, which attempt to predict performance using only the initial state of the network (e.g., using the Neural Tangent Kernel). However, Mok et al. discovered that these metrics can be unreliable due to "kernel twisting."

The authors argue that a small computational investment—training for just a few epochs—is significantly more effective because it captures the dynamic adaptation of the network. This brief period of learning allows the model to move past its random initialization, providing a much more accurate guide for the search process than static analysis alone.

Applications in Automated Engineering

These proxies are the engine behind modern automated Machine Learning Engineering (MLE). Yuan et al. proposed the ArchPilot framework, which uses a weighted combination of proxies—such as single-epoch validation and data subsets—to guide multi-agent systems through complex search spaces.

By treating evaluation as a budget-constrained estimation problem, these methods transform architecture design from a "dark art" (as described by Sun et al.) into a scalable engineering discipline. Whether through genetic algorithms (Lemieux et al.), reinforcement learning (Tian et al.), or active learning (Geifman et al.), the use of low-fidelity proxies ensures that the most promising models are identified with a fraction of the traditional computational cost.

Key Foundational Work

Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data (2017) — contributed 2 concept elements to this direction.
Genetic engineering of handwriting representations (2002) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_fidelity_proxy_evaluation_a13e2d05,
  title = {low-fidelity proxy evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-fidelity-proxy-evaluation/rdr_a13e2d0515d74e60b0b2f6ba5b2d0998},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_fidelity_proxy_evaluation_a13e2d05_v1,
  title = {low-fidelity proxy evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-fidelity-proxy-evaluation/rdr_a13e2d0515d74e60b0b2f6ba5b2d0998/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake