Neural Architecture Search (NAS) was originally envisioned as a way to automate the design of deep learning models, but early methods were computationally ruinous, often requiring thousands of GPU days to find a single optimal structure. To solve this, researchers moved toward "One-Shot" NAS. Instead of training thousands of candidate models from scratch, they train a single, massive Super-network (or Supernet) that contains all possible architectures as sub-networks.
By sharing weights across these sub-networks, the search process is dramatically accelerated. For instance, Zhaohui Yang and Xinghao Chen demonstrated with CARS that this approach can slash search times to as little as 0.4 GPU days. However, this efficiency comes with a significant catch: the performance of a sub-network within the Supernet often fails to predict how that same network will perform when trained independently.
The Weight-Sharing Proxy
The core intuition of One-Shot NAS is that the Supernet acts as a performance proxy. If a specific path through the Supernet (a sub-network) achieves high accuracy using the shared weights, we assume it will also be a high-performer when trained from scratch.
As Ren Pengzhen notes in A Comprehensive Survey of Neural Architecture Search, this "recycled" strategy is what makes modern NAS feasible. But the assumption of a reliable proxy is frequently violated. In practice, the weights that are optimal for the Supernet as a whole are rarely optimal for any individual sub-network. This leads to what Yongqi Zhang and quanming yao describe as rank disorder: the ranking of architectures based on shared weights does not correlate with their true standalone ranking.
Why Rank Correlation Fails
The lack of rank correlation is the primary bottleneck in one-shot methods. Several factors contribute to this "ranking bias" or "evaluation degradation":
- Co-adaptation: Because weights are shared, different sub-networks "interfere" with each other during training. A sub-network might look good only because it is piggybacking on weights optimized for a different, more dominant path.
- Multi-model Forgetting: As Huiqi Li explains in One-Shot Neural Architecture Search via Novelty Driven Sampling, the Supernet may "forget" the optimal weights for one sub-network while being updated for another, leading to deceptive rewards.
- Space Explosion: When the search space is massive, the Supernet cannot possibly train all sub-networks to a state where their shared weights are meaningful. Yaofo Chen refers to this in Breaking the Curse of Space Explosion as a gap that causes search algorithms to select suboptimal designs that simply happened to benefit from the training dynamics.
Strategies for Improving Correlation
To make One-Shot NAS reliable, researchers have developed several mechanisms to "smooth" the search space or improve the quality of the performance proxy.
Modeling the Search Space
Instead of trusting the raw accuracy of sub-networks, some methods attempt to model the underlying relationship between architecture and performance. Lingxi Xie and Xin Chen proposed using Graph Convolutional Networks (GCNs) to fit the search space. By treating architectures as graphs, the GCN can "smooth out" the random noise in shared-weight estimates, leading to more consistent rankings.
Better Performance Estimators
Other researchers look for proxies that are more stable than final validation accuracy. Clare Lyle and Yarin Gal introduced Training Speed Estimation (TSE), which uses the sum of early training losses. They found that how quickly a model learns in the early stages is often a better predictor of its final standalone performance than its accuracy within a weight-sharing Supernet.
Curriculum and Pruning
Rather than searching the entire space at once, methods like TNAS, developed by Guocheng Qian and Guohao Li, use tree-based structures to prune unpromising branches early. Similarly, Curriculum NAS starts with a simple search space and gradually increases complexity, ensuring the Supernet learns a stable ranking before the space becomes too large to manage.
The Path Forward
The transition from expensive, discrete searches to efficient one-shot searches has made NAS accessible for specialized tasks, such as Acoustic Scene Classification and Robot Vision. However, the success of these applications depends entirely on bridging the rank correlation gap. Whether through better sampling strategies, more robust predictors, or sophisticated search space modeling, the goal remains the same: ensuring that the "shortcut" of weight-sharing doesn't lead the searcher to a dead end.