Ranking Based Surrogate Models for Architecture Search

This direction is merged into another direction: computational cost of architecture search, relative performance prediction, relative performance ranking. Its last version stays readable and citable here.

Version 1 of 1135 papers236 conceptsfrozen 2026-10-04

Neural Architecture Search (NAS) is fundamentally a selection problem. To find the best model in a massive search space, we don't actually need to know if an architecture achieves 76.2% or 76.4% accuracy; we only need to know that the second one is better than the first.

Traditional performance predictors—often called surrogate models—treat this as a regression task, attempting to map an architecture's encoding to an absolute accuracy value. However, a growing body of research suggests that this "absolute" approach is unnecessarily difficult and prone to noise. Instead, the field is shifting toward ranking-based surrogates, which prioritize the relative order of candidates over their exact performance metrics.

The Regression Trap

In standard predictor-based NAS, researchers often use Mean Squared Error (MSE) to train surrogates. As noted by Shenghe Zheng and colleagues in DCLP, regression objectives struggle when performance gaps between architectures are narrow. A small numerical error in an absolute prediction can lead to a total "rank disorder," where a search algorithm accidentally discards a top-tier model because its predicted score was slightly lower than a mediocre one.

Furthermore, absolute accuracy is a moving target. It depends on the dataset, the training recipe, and the hardware. Xian-Rong Zhang and Yue-Jiao Gong argue in DCL-ENAS that predicting exact accuracy is a "difficult regression task prone to noise," whereas identifying which of two models is superior is a much more stable objective.

Learning to Compare

The core mechanism of this research direction is the pairwise ranking loss. Instead of training on $(architecture, accuracy)$ pairs, the model is trained on $(architecture_A, architecture_B, label)$, where the label indicates which model performed better.

This shift offers several mathematical and practical advantages:

  1. Data Efficiency: If you have 100 trained architectures, you only have 100 data points for regression. However, those same architectures can generate nearly 5,000 unique pairwise comparisons. This combinatorial explosion allows methods like ReNAS to achieve high ranking accuracy using as little as 0.1% of the search space as training data.
  2. Simplified Objectives: In many cases, the surrogate can be simplified into a binary classifier. Yuxi Huan and Fan Wu 0009 demonstrate in IEO that a simple classifier can predict whether an "offspring" architecture will outperform its "parent," allowing the search to skip the training of likely underperformers.
  3. Siamese Architectures: Many ranking predictors, such as those in RANK-NOSH, use Siamese Networks. These networks process two architectures through shared weights and use a hinge loss or Bradley-Terry model to determine the winner.

Robustness to Partial Information

Ranking is particularly effective when dealing with "proxy" evaluations—shortened training runs or weight-sharing supernets that don't reflect final accuracy but do preserve the relative order of models.

Ruochen Wang and Minhao Cheng show in RANK-NOSH that ranking is more robust than regression when comparing models trained for different numbers of epochs. Similarly, Peijie Dong and Xin Niu use a pairwise ranking loss in PGONAS to anchor weight-sharing models against "ranking drift," ensuring that the performance of a sub-network within a giant graph matches its true performance when trained alone.

Prioritizing the "Gems" over "Junk"

A critical insight from Yuge Zhang and the AceNAS team is that not all ranks are equal. In a search space of millions, we don't care if the predictor correctly ranks the 1,000,000th best model against the 1,000,001st. We only care about the ranking at the very top of the distribution.

By using metrics like Normalized Discounted Cumulative Gain (NDCG)—the same metric used by search engines to rank web results—researchers can train surrogates that are hyper-accurate for high-performing "gems" while ignoring the relative order of "junk" architectures. This philosophy is echoed in RoBoT, where Zhenfeng He and Yao Shu argue that search efficiency depends on proxy precision at the top of the ranking, rather than global correlation across the entire space.

Multi-Objective and Pareto Ranking

The ranking paradigm extends naturally to multi-objective search (e.g., balancing accuracy vs. latency). In CENAS, Lianbo Ma and colleagues replace accuracy predictors with a Pareto-wise ranking classifier. Instead of predicting two separate numbers, the model predicts whether one architecture "dominates" another in the Pareto sense. This approach, also explored by Yaofo Chen in PNAG, allows the search to navigate the trade-off frontier without ever needing to calibrate absolute performance values across different hardware constraints.

Papers in this version

Showing 60 of 135 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ranking_based_surrogate_models_for_archi_28cdc8fe,
  title = {Ranking Based Surrogate Models for Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ranking-based-surrogate-models-for-architecture-search/rdr_28cdc8fe361040cc9d5d634373f5f051},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_ranking_based_surrogate_models_for_archi_28cdc8fe_v1,
  title = {Ranking Based Surrogate Models for Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ranking-based-surrogate-models-for-architecture-search/rdr_28cdc8fe361040cc9d5d634373f5f051/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}