The central challenge of Neural Architecture Search (NAS) is not just finding a good architecture, but doing so without training thousands of candidates to completion. To avoid this "billion-dollar" compute bill, researchers use performance proxies: cheap shortcuts that estimate how well a model will eventually perform.
However, a recurring theme across recent research is that these shortcuts are often deceptive. This phenomenon, known as the correlation gap, occurs when the ranking of architectures provided by a proxy does not match their true ranking after full training. If a proxy cannot reliably distinguish a "winner" from a "loser," the search process becomes no better than random selection.
The Proxy Dilemma: Speed vs. Fidelity
In an ideal world, we would train every candidate architecture in our search space. In reality, we use lower-fidelity evaluations to save time. As noted in Neural Architecture Search for Dense Prediction Tasks in Computer Vision, these proxies typically involve training on reduced datasets, using fewer iterations, or shrinking the model size.
The danger is that these "miniature" evaluations often fail to capture the long-term learning dynamics of a full-scale model. Xuefei Ning and colleagues in Evaluating Efficient Performance Estimators of Neural Architectures demonstrate that many estimators exhibit significant biases—for instance, favoring smaller models that learn quickly but plateau early. This "fidelity gap" means that the architectures that look best at epoch 5 are rarely the ones that are best at epoch 200.
The Weight-Sharing Trap
The most popular "speed hack" in NAS is weight-sharing (or one-shot NAS). Instead of training models independently, researchers train a single "supernet" containing all possible operations. Each candidate architecture is then evaluated as a sub-network of this supernet.
While efficient, this approach introduces interference. Jin Xu explains in Analyzing and Mitigating Interference in Neural Architecture Search that different sub-models compete for the same weights, leading to inaccurate performance estimates. Because the shared weights must "please everyone," they end up representing no single architecture well. This leads to what Guohao Li calls "degenerate correlation" in SGAS: Sequential Greedy Architecture Search, where architectures that appear effective within the supernet perform poorly when evaluated as standalone models.
Zero-Cost Proxies and the Search for Better Metrics
To bypass training entirely, a new frontier of zero-cost proxies has emerged. These metrics, such as the SWAP-Score introduced by Yameng Peng in SWAP-NAS, attempt to predict performance using only a single forward pass of data at initialization. Other methods, like RMI-NAS by Xiang Fei, use Representation Mutual Information to see how well a candidate's internal features align with a reference model.
While these methods can search CIFAR-10 in minutes, they are notoriously brittle. Zhenfeng He and Yao Shu argue in Robustifying and Boosting Training-Free Neural Architecture Search that individual zero-cost metrics often fail to generalize across different tasks. A metric that works for image classification might show a negative correlation for medical imaging or quantum feature maps, as explored in QuProFS.
Bridging the Gap: Ensembles and Few-Shot Learning
To fix these unreliable rankings, researchers are moving toward more robust, hybrid evaluation strategies:
- Adaptive Normalization: Bailin Li showed in EagleEye that simply re-calibrating Batch Normalization statistics for each sub-network can dramatically improve the correlation between proxy scores and final accuracy.
- Few-Shot NAS: Instead of one supernet for everything, Yiyang Zhao and Yuandong Tian propose Few-Shot NAS, which splits the search space into multiple smaller supernets. This reduces weight interference and provides a much more accurate ranking of candidates.
- Dynamic Ensembling: Recognizing that no single proxy is perfect, Junbo Zhao developed DELE, which uses a gating network to blend predictions from multiple low-fidelity experts. This allows the system to rely on different proxies at different stages of the search.
Ultimately, the field is shifting away from "fastest possible" search toward "reliably fast" search. As Aloïs Pourchot concludes in To Share or Not To Share, a high global correlation is meaningless if the proxy fails to rank the very best models correctly. The goal is no longer just to find a proxy that is "good enough" on average, but one that is precise at the top of the distribution.