Neural Architecture Search (NAS) promises to automate the tedious process of designing deep learning models. However, training thousands of candidate networks from scratch to find the best one is computationally prohibitive. "One-shot" or weight-sharing NAS solves this by training a single massive "supernet" that encompasses the entire search space. Every individual architecture is simply a sub-network within this supernet, inheriting its weights directly.
While computationally elegant, this paradigm introduces a fundamental flaw: the performance of a sub-network using shared weights often correlates poorly with its true performance when trained independently from scratch. As demonstrated by Chu et al. and Jeon et al., the success of a supernet depends not on its absolute top-1 accuracy, but on its ranking consistency—its ability to correctly preserve the relative order of sub-networks. Without reliable rankings, the search phase becomes a lottery.
The Root Cause: Interference and Forgetting
Why do internal rankings degrade? When a supernet optimizes its weights for one sub-network, it inadvertently overwrites or disrupts the weights needed by another. This phenomenon is frequently framed as "performance disturbance" or "multi-model forgetting."
Xu et al. identify gradient conflict—where different sub-networks demand opposing updates to the same shared parameters—as a primary driver of ranking disorder. When these updates clash, the supernet becomes an unreliable proxy evaluator. This can lead to "model collapse," a failure mode where the search algorithm heavily favors simple operations (like skip-connections) over complex convolutions merely because they stabilize faster during the chaotic training phase, as noted by Zhou et al..
To make weight-sharing NAS viable, researchers have developed several distinct strategies to protect and align the supernet's internal ranking signal.
Strategy 1: Anchoring and Distillation
One direct way to enforce ranking consistency is to anchor the supernet's internal predictions to known reference points or external proxies.
Standard Supernet: [Subnet A] ──(Unstable Weights)──> Predicted Rank: #1 (True Rank: #5)
Anchored Supernet: [Subnet A] ──(Aligned via Anchors)─> Predicted Rank: #5 (True Rank: #5)
▲
[Landmark Models] (Ground Truth Reference)
- Landmark Regularization: Yu et al. introduce a method that uses a small set of pre-evaluated "landmark" architectures. By forcing the supernet to maintain the correct relative rankings of these landmarks during training, the entire internal weight space is anchored to ground-truth performance.
- Ranking Distillation: Instead of training landmarks, Dong et al. leverage cheap, zero-cost proxies (such as FLOPs or parameter counts) as teachers. They distill this ranking knowledge into the supernet via a specialized sampler, significantly reducing ranking disorder.
- Internal Self-Distillation: Peng et al. utilize high-performing internal paths within the supernet as "teachers" to guide the training of other sampled sub-networks, boosting the overall statistical correlation between shared-weight and standalone performance.
Strategy 2: Mitigating Gradient Interference
Instead of correcting the rankings after the fact, another branch of work prevents sub-networks from sabotaging each other during gradient updates.
- Orthogonal Gradients: Niu et al. propose a disturbance-immune strategy that utilizes orthogonal gradient descent. When updating weights for a given architecture, the gradients are projected so they do not degrade the performance of previously evaluated paths.
- Meta-Learning: Ha et al. reframe supernet training through the lens of Model-Agnostic Meta-Learning (MAML). By treating each sub-network architecture as a distinct "task," the supernet is forced to learn unbiased, meta-features that generalize cleanly across the entire search space rather than overfitting to a subset of paths.
- Diversity Maximization: To combat multi-model forgetting, Li et al. explicitly maximize architectural diversity during the sampling process, which stabilizes the training environment and yields higher Kendall's Tau correlation metrics.
Strategy 3: Fair and Dynamic Optimization
Standard supernet training can be inherently unfair, favoring architectures that are sampled more frequently or those that optimize quickly.
- Strict Fairness: Chu et al. address this with FairNAS, a framework that enforces strict fairness by ensuring every choice in the search space receives equal optimization opportunities in each step, preventing the supernet from developing early, unearned biases.
- Subnet-Aware Dynamics: Recognizing that different architectures have varying capacities, Jeon et al. dynamically adjust learning rates based on model complexity and isolate momentum buffers for different architecture groups, ensuring balanced optimization.
- K-Shot Weight Sharing: Breaking the rigid constraint of a single shared weight matrix, Su et al. maintain a dictionary of $K$ distinct supernets. A master "simplex-net" adaptively combines these weight sets for each specific architecture, drastically reducing weight-coupling artifacts.
Strategy 4: Search Space Pruning and Fine-Tuning
A final approach to improving ranking fidelity involves narrowing down the search space or adaptively shifting the supernet's focus toward high-potential candidates.
- Progressive Pruning: Xin Xia demonstrates that progressively removing redundant or poorly performing operations directly from the search space minimizes weight coupling among the remaining candidates, making the final evaluation phase significantly cleaner.
- Greedy Filtering: Similarly, You et al. introduce GreedyNAS, which filters out weak paths during training to focus supernet updates primarily on top-tier candidates, improving fine-grained ranking capability where it matters most.
- Supernet Shifting: For fine-grained comparisons among the absolute best performers, Zhang et al. introduce an adaptive fine-tuning approach that accumulates training loss on frequently sampled, superior architectures, refining the supernet's order-preserving capabilities during the final stages of the search.
The underlying insight across all these methodologies is identical: a supernet does not need to achieve perfect standalone accuracy during the search phase. It simply needs to be a fair, consistent, and reliable judge. This core principle extends beyond NAS; for instance, Li et al. demonstrate that maintaining a tight rank correlation via adaptive batch normalization is equally vital when evaluating sub-networks for efficient neural network pruning.
Key Foundational Work
K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets (2021) — contributed 2 concept elements to this direction.
Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search (2021) — contributed 2 concept elements to this direction.
Disturbance-immune Weight Sharing for Neural Architecture Search (2020) — contributed 1 concept elements to this direction.