Neural Architecture Search (NAS) promises to automate the design of deep learning models, but its early iterations were prohibitively expensive, requiring thousands of GPU days to train and evaluate candidate architectures from scratch. The breakthrough that made NAS practical was weight-sharing, a technique where millions of potential architectures are nested within a single, massive "supernet."
In this paradigm, every candidate architecture is a sub-network that inherits its weights directly from the supernet. By training the supernet once, researchers hoped to "sample" high-performing models instantly. However, as Kaicheng Yu and Christian Sciuto demonstrated in Evaluating the Search Phase of Neural Architecture Search, these efficient methods often struggle to outperform simple random search. The culprit is a phenomenon known as rank disorder.
The Supernet Deception
The fundamental assumption of weight-sharing is that if Sub-network A performs better than Sub-network B using shared weights, it will also perform better when both are trained independently from scratch.
In practice, this assumption frequently fails. A sub-network's performance within a supernet often has little to no correlation with its true standalone potential. This "optimization gap," as described by Lingxi Xie and Xin Chen 0033 in Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap, occurs because the shared weights are forced to be "jacks of all trades." They are optimized to function across a broad range of architectures simultaneously, leading to co-adaptation and interference.
Several factors contribute to this ranking inconsistency:
- Size Bias: Renqian Luo and Enhong Chen found that supernets unfairly favor small, easy-to-train models over larger, more capable ones that require more optimization to show their potential.
- Operation vs. Topology Coupling: In differentiable NAS (DARTS), the search for what a layer does is often tangled with where it connects. Yuchao Gu and Lijuan Wang argue in DOTS that this coupling makes architecture weights unreliable indicators of actual utility.
- Posterior Fading: As the search space grows, shared weights become less representative of any single model, a problem Xiang Li identifies as "posterior fading."
Restoring the Rank Correlation
To make one-shot NAS reliable, researchers have developed several strategies to "denoise" the performance signal from the supernet.
Balanced and Novelty-Driven Sampling
If the supernet is biased toward certain paths, the solution is to change how we sample. Renqian Luo introduced Balanced NAO, which samples architectures in proportion to their size to ensure larger models receive adequate training. Alternatively, Huiqi Li and Miao Zhang 0022 proposed EN²AS, which ignores unreliable accuracy scores during training entirely, instead sampling based on novelty to prevent the supernet from getting stuck in local optima.
Anchoring with Landmarks
One of the most intuitive solutions is to provide the supernet with a "ground truth" reference. Kaicheng Yu introduced Landmark Regularization, which uses a small set of architectures with known standalone performance to anchor the supernet’s internal rankings. By forcing the supernet to correctly rank these "landmarks," the search process becomes significantly more stable.
Beyond Parameter Magnitude
In many NAS methods, the "importance" of an operation is judged by the magnitude of its learned weights. However, Han Xiao 0010 and Ziwei Wang argue in Shapley-NAS that magnitude is a poor proxy for contribution. They instead use Shapley values—a concept from cooperative game theory—to measure the true marginal contribution of each operation to the model's success, leading to much more accurate architecture selection.
The Path Forward
The evolution of NAS is moving away from simply searching faster and toward searching more accurately. Whether through sequential pruning (Guohao Li), geometry-aware updates (Liam Li), or using Graph Convolutional Networks to denoise the search space (Lingxi Xie), the goal remains the same: closing the gap between the supernet's predictions and the reality of standalone training.
As summarized by Colin White in Neural Architecture Search: Insights from 1000 Papers, the field has shifted from a "black box" optimization problem to a principled engineering discipline focused on the reliability of performance estimation.