When optimizing machine learning pipelines or searching for neural architectures, the traditional instinct is to build a precise regression model. We want to predict exactly how well a candidate architecture will perform—estimating its final validation accuracy to the nearest decimal point. However, this absolute precision comes at an astronomical computational cost, often requiring thousands of fully trained models to calibrate the predictor.
The core insight driving a powerful paradigm shift in model selection is simple: we do not need to know a model's absolute performance; we only need to know if it is better than its competitors. As noted by Francesco Strino et al. in VDA, a Method of Choosing a Better Algorithm with Fewer Validations, the primary goal in practical optimization is identifying the superior tool under a fixed budget, not global accuracy estimation. By shifting the objective from absolute value regression to relative performance ranking, we unlock the ability to use ultra-low-cost proxies that drastically accelerate search spaces.
The Futility of Absolute Regression
Predicting the exact accuracy or latency of a diverse set of neural networks across different hardware targets is inherently noisy and complex. Andrew Brock et al. point out in SMASH: One-Shot Model Architecture Search through HyperNetworks that the massive computational overhead of obtaining absolute performance metrics is entirely unnecessary if a cheaper proxy can accurately preserve the ranking of different designs.
Furthermore, the authors of DetOFA: Efficient Training of Once-for-All Networks for Object Detection using Path Filter emphasize that attempting to predict exact accuracy values across diverse tasks introduces unnecessary noise. Instead, focusing on whether one architecture outperforms another within a similar resource envelope provides a much more robust signal for optimization.
Exploiting Low-Fidelity Proxies
If the goal is purely rank preservation, we can dramatically lower the fidelity of our evaluations. Instead of training every candidate architecture to full convergence, we can use highly abbreviated or altered training regimes.
In their comprehensive survey, Thomas Elsken et al. detail in Neural Architecture Search: A Survey how performance estimation strategies can prioritize relative ranking by training on data subsets or for fewer epochs. This stability of relative rankings during abbreviated training is leveraged by Han Zhou et al. in AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning and by Hao Tan et al. in RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning to evaluate complex configurations in a fraction of the time.
Taking this to the extreme, Shengran Hu et al. introduce Random-Weight Evaluation in Accelerating multi-objective neural architecture search by random-weight evaluation, demonstrating that freezing a network with random weights and training only the final classification layer yields a proxy that maintains a high Spearman rank correlation with ground-truth performance. Similarly, Xiawu Zheng et al. show in Rethinking Performance Estimation in Neural Architecture Search that maximizing the correlation between low-cost proxy results and ground-truth performance allows algorithms to achieve a 1,000x speedup in search time with negligible loss in final accuracy. This focus on fast estimators that maintain high correlation with final accuracy is echoed by Bailin Li et al. in EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning.
This principle extends beyond accuracy to hardware metrics like latency. Bingqian Lu et al. exploit "latency monotonicity" in One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search—the fact that relative model speeds remain highly correlated across different physical devices—proving that a single proxy device is sufficient to rank and select Pareto-optimal models for entirely different deployment targets.
Designing Rank-Aware Controllers and Surrogates
To effectively navigate massive search spaces, automated search algorithms rely on surrogate models or controllers. When these surrogates are trained to predict relative rankings rather than absolute values, their sample efficiency skyrockets.
Chenxi Liu 0001 et al. show in Progressive Neural Architecture Search that surrogate models do not require high precision in absolute metrics to focus training resources on promising configurations. In MemNAS: Memory-Efficient Neural Architecture Search With Grow-Trim Learning, Peiye Liu et al. demonstrate that a ranking-based controller provides a more reliable signal for steering search processes, especially when performance differences between candidates are subtle.
To further refine this, Yiran Chen 0001 proposes predicting the accuracy difference between pairs of architectures in Delta-NAS: Difference of Architecture Encoding for Predictor-Based Evolutionary Neural Architecture Search, which achieves a significantly higher ranking correlation than predicting absolute values. Other approaches, such as the context-aware predictors by Han Ji et al. in CAP: A Context-Aware Neural Predictor for NAS or the Graph HyperNetworks by Chris Zhang et al. in Graph HyperNetworks for Neural Architecture Search, align their training objectives directly with ranking to identify top-tier architectures without requiring full convergence.
This mechanism is similarly utilized by Wenhao Li et al. in DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architectures to rapidly filter candidates based on early-stage indicators. Ultimately, shifting surrogate modeling toward ranking-based objectives fundamentally mitigates the noise and inaccuracies inherent in absolute value regression, as highlighted by Jia Liu et al. in Bi-fidelity Evolutionary Multiobjective Search for Adversarially Robust Deep Neural Architectures.
Rethinking Evaluation: Focus on the Top Tier
If the purpose of a proxy is to find the best model, how should we evaluate the proxy itself? Traditionally, researchers used global correlation metrics like Kendall’s Tau or Spearman’s Rho across the entire search space.
However, Ondrej Týbl et al. argue in Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights that global metrics can be deeply misleading. In practice, we do not care if a proxy incorrectly ranks two terrible architectures; we only care about its ability to correctly identify and rank the highest-performing architectures. By evaluating proxies using Normalized Discounted Cumulative Gain (nDCG), the authors prioritize top-tier accuracy selection over global distribution matching, providing a much more relevant signal for practical search tasks.
Generalization Across Domains
The power of relative ranking extends far beyond standard neural architecture search:
- Low-Resource NLP: Żeljko Agić et al. demonstrate in Cross-lingual tagger evaluation without test data (EACL (2) 2017) that proxy metrics based on simple dictionary validity consistently preserve the performance ranking of cross-lingual taggers, allowing researchers to select superior models without any gold-standard human annotations.
- Data-Limited Simulations: Ghadi S. Al Hajj et al. show in Meta simulation approach for evaluating machine learning method selection in data limited settings (Scientific 2025) that effective model selection in simulated environments depends entirely on the simulation's ability to preserve relative algorithm rankings rather than mirroring absolute real-world metrics.
- Hardware Design Space Exploration: In automated hardware synthesis, Bharath Srinivas Prabakaran notes in ApproxFPGAs: Embracing ASIC-Based Approximate Arithmetic Components for FPGA-Based Systems that a lack of high-fidelity ranking in performance estimation models prevents designers from accurately identifying the true Pareto-optimal frontier.
- Robustness: Shengcao Cao et al. show in Neighborhood-Aware Neural Architecture Search that aggregating the relative performance of an architecture and its structural neighbors prevents the selection of brittle models that only look good in isolation.
Key Foundational Work
VDA, a Method of Choosing a Better Algorithm with Fewer Validations (2011) — contributed 1 concept elements to this direction.
Cross-lingual tagger evaluation without test data (2017) — contributed 1 concept elements to this direction.