Neural Architecture Search (NAS) has evolved from a computationally expensive brute-force search into an efficient "train-once, sample-many" paradigm. At the heart of this shift is the weight-sharing supernet: a single, massive network containing billions of potential sub-architectures (subnets) that all share the same set of parameters.
While this approach drastically reduces training costs, it introduces a fundamental optimization challenge known as gradient interference. When multiple architectures compete for the same weights, the optimization process becomes a tug-of-war that can degrade performance and destroy the reliability of the search.
The Weight Coupling Problem
In a standard neural network, gradients point toward a direction that minimizes the loss for that specific architecture. In a supernet, however, a single parameter might participate in thousands of different paths.
As Jin Xu and Xu Tan explain in their work on Analyzing and Mitigating Interference in Neural Architecture Search, different candidate models often produce conflicting gradient updates. This "interference" prevents the shared weights from converging to a state that accurately represents the true performance of any individual subnet.
This phenomenon, often called weight coupling, leads to a "ranking inconsistency" problem: a subnet that performs well within the supernet might perform poorly when trained from scratch, and vice versa. Peijie Dong and colleagues note in Prior-Guided One-shot NAS that this coupling makes it nearly impossible for the supernet to reach a state that is optimal for all constituent sub-networks simultaneously.
Why Gradients Conflict
The intuition behind interference lies in the diverse requirements of different model scales.
- Scale Disparity: Chen Tang and Li Lyna Zhang observe in ElasticViT that updates for small sub-models often pull shared parameters in directions that are detrimental to larger architectures.
- Topological Distance: Research by Jin Xu shows that as the number of differing operators between two models increases, the cosine similarity of their gradients decreases.
- Precision Conflicts: In mixed-precision settings, Masakazu Yoshimura found that the optimization requirements of low-precision subnets frequently clash with those of high-precision ones (Mixed-precision Supernet Training).
Even in Multi-Task Learning, as Steven Y. Guo points out, forcing a shared encoder to optimize for all objectives simultaneously creates non-orthogonal gradients that compete for the same parameter space (On Steering Multi-Annotations per Sample).
Strategies for Mitigation
Researchers have developed several distinct approaches to manage these conflicting signals.
1. Pruning the Search Space
One of the most effective ways to reduce interference is to simply reduce the number of competing architectures. Haibin Wang and Ce Ge introduced PreNAS, which uses "zero-cost" proxies to identify high-quality "preferred" architectures before training begins. By focusing the supernet's training on this elite subset, they eliminate updates from poor-performing models that would otherwise degrade the convergence of superior candidates. Similarly, CompOFA uses compound scaling heuristics to prune sub-optimal architectures, slashing training time and interference.
2. Architectural Decoupling
If the problem is that subnets share too much, the solution is to give them their own "space." Ganesh Jawahar and Haichuan Yang proposed Mixture-of-Supernets (MoS), which uses a Mixture-of-Experts (MoE) approach to generate architecture-specific weights. This replaces strict weight sharing with flexible, router-based weight generation, allowing the model to adapt to diverse requirements without destructive interference.
3. Dynamic Training and Scheduling
Other methods focus on how and when subnets are updated.
- Delayed Introduction: Alind Khare introduced DεpS, which delays the introduction of smaller subnets until the main model is partially trained, preventing early-stage gradient spikes from destabilizing the global weights.
- Momentum Separation: Jeimin Jeon and Bumsub Ham found that aggregating disparate gradients into a single shared momentum buffer introduces noise. Their Subnet-Aware Dynamic Supernet Training strategy separates momentum buffers for different architecture groups.
Is Interference the Only Culprit?
While the consensus highlights interference, some researchers suggest a more nuanced view. Jordan Shipard and Arnold Wiliem argue in Does Interference Exist When Training a Once-For-All Network? that sampling bias—the frequency and order in which subnets are seen—might be as significant as weight interference. They show that simple Random Subnet Sampling (RSS) can achieve high accuracy by ensuring a more balanced distribution of updates, suggesting that the granularity of subnet switching is a critical lever in avoiding weight conflicts.
Ultimately, the field is moving toward a "conflict-aware" training paradigm. Whether through smarter sampling (PA&DA), gradient matching (A Survey of Supernet Optimization), or architectural partitioning, the goal remains the same: ensuring that the shared weights of a supernet are a collaborative foundation rather than a site of architectural competition.