To deploy deep learning models across a heterogeneous landscape of hardware—from edge sensors to cloud servers—the machine learning community increasingly relies on "Once-For-All" (OFA) networks and One-Shot Neural Architecture Search (NAS). Instead of training thousands of distinct models from scratch, we train a single, massive supernet. By activating different subsets of this supernet, we can instantly extract tailored subnets for any device profile.
It is an elegant idea, but it introduces a severe optimization pathology: weight-sharing interference. When a single set of parameters must simultaneously satisfy the representational needs of millions of diverse architectures, the optimization process becomes a destructive tug-of-war.
The Geometry of Gradient Conflict
Why does weight sharing degrade performance? The root cause lies in the geometry of the loss landscape during joint optimization.
When we sample a specific subnet to process a batch of data, it computes a gradient update $g_1$ tailored to its unique architecture. If we immediately or simultaneously sample a different subnet, it computes an update $g_2$. If these subnets require fundamentally different feature representations from the same parameters, their gradients will point in conflicting directions, meaning their dot product is negative ($g_1 \cdot g_2 < 0$).
Gradient g1 (Subnet A)
^
\
\
+-----> Gradient g2 (Subnet B)
/
/
v
Conflicting Updates (g1 · g2 < 0)
As Xu et al. demonstrated, this gradient interference is highly correlated with the topological distance between subnets. The more two architectures differ in their operators or structural paths, the lower the cosine similarity of their gradients, leading to highly conflicting parameter updates.
When multiple subnets are updated within the same training step (intra-batch switching), these conflicting forces cancel each other out or introduce massive noise. Shipard et al. showed that this intra-batch interference destabilizes the shared weights, preventing the supernet from finding a single parameter set that works for diverse sub-architectures.
The Consequences: Ranking Disorder and Capacity Entanglement
This interference breaks weight-sharing systems in two distinct ways:
1. Ranking Disorder
The fundamental assumption of one-shot NAS is that a subnet's performance within the supernet serves as a reliable proxy for its performance when trained independently. However, because weights are constantly pulled around by competing subnets, parameters co-adapt to satisfy the average requirement of the search space rather than the specific needs of any single architecture. As Dong et al. and Yu et al. note, this co-adaptation causes severe ranking disorder, meaning the supernet's internal performance metrics fail to correlate with standalone accuracy.
2. Capacity Entanglement
Forcing a massive model and a highly compressed model to share parameters creates an asymmetric performance trade-off. Ao et al. observed that the largest configurations often suffer an accuracy penalty because their weights must remain flexible enough to accommodate the constrained representations of smaller subnets. Conversely, He et al. highlighted that different model scales require distinct feature representations, meaning a naive update for one scale directly degrades another.
Mitigating Interference: How to Stabilize Supernets
To unlock the true potential of once-for-all networks, researchers have developed several mechanisms to isolate, correct, or bypass conflicting gradients.
Temporal Separation and Scheduling
Instead of forcing the network to digest conflicting gradients simultaneously, we can separate them in time. Shipard et al. proposed Random Subnet Sampling (RSS), which switches subnets at the epoch level rather than the batch level. This temporal separation allows the model to stabilize weights for a specific configuration before switching.
Similarly, Annavajjala et al. introduced Delayed $\epsilon$-Shrinking (D$\epsilon$pS). Fully training a large supernet first causes weights to specialize, making it harder for smaller subnets to adapt. D$\epsilon$pS delays the introduction of smaller subnets until the model is partially trained, then gradually ramps up their learning rates to mitigate gradient spikes and ensure a smooth transition for global weights.
Pruning the Conflict Space
We can minimize interference by simply reducing the structural diversity of competing architectures. Sahni et al. (CompOFA) use compound scaling heuristics to prune poorly performing or structurally extreme architectures from the search space entirely. By not forcing weights to accommodate sub-optimal architectures, the remaining high-quality models achieve better final accuracy. You et al. (GreedyNAS) similarly filter out weak candidate paths during training to focus shared weight capacity on the most promising paths.
Optimization Isolation
If gradients must conflict, we can prevent them from polluting the global optimization history. Momentum tracking aggregates historical gradient directions to smooth out optimization. However, when disparate gradients from different subnets are thrown into a single momentum buffer, it introduces severe noise.
Jeon solved this by separating momentum buffers for different architecture groups, ensuring that a subnet's update history does not distort the trajectory of a completely different architecture group. For explicit intervention, Yang et al. introduced Alternating Sparse Training with an explicit gradient correction step to actively realign conflicting updates across different sparsity levels.
Standard Supernet: [Subnet A Gradient] \
[Subnet B Gradient] ---> [Single Momentum Buffer] ---> Destructive Noise
[Subnet C Gradient] /
Isolated Supernet: [Subnet A Gradient] ---> [Buffer A] \
[Subnet B Gradient] ---> [Buffer B] ---> Stable, Targeted Updates
[Subnet C Gradient] ---> [Buffer C] /
Relaxing Strict Weight Sharing
Finally, we can soften the constraint that every subnet must use the exact same parameters. Su et al. (K-shot NAS) maintain a dictionary of multiple supernets and learn to adaptively combine their weights for specific paths.
Taking this further, the authors of the Mixture-of-Supernets framework utilize an architecture-routed Mixture-of-Experts (MoE) layer to generate architecture-specific weights. By replacing rigid parameter sharing with a router that generates weights based on a model's specific dimensions, they ensure that training signals for small models are routed away from, and do not interfere with, the parameters of larger models.