Neural Architecture Search (NAS) promises to automate the design of deep learning models, but its early iterations were notoriously expensive, often requiring thousands of GPU days. To solve this, researchers moved toward weight-sharing or one-shot NAS. Instead of training thousands of models from scratch, we train a single, massive "supernet" that contains every possible architecture in the search space as a sub-network. Once the supernet is trained, we simply "sample" sub-networks and evaluate them using the shared weights.
However, this efficiency introduces a new problem: the optimization gap. As Lingxi Xie and colleagues note in Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap, a supernet often fails to accurately rank its sub-architectures. A model that performs well within the supernet might fail when trained independently. The root of this deception is unfairness during training.
The Matthew Effect in Architecture Search
In a standard weight-sharing setup, we train the supernet by randomly sampling paths. However, not all paths are created equal. Some operations, like skip connections or small $3 \times 3$ convolutions, converge much faster than complex ones.
As Weijun Hong and Guilin Li describe in DropNAS, this creates a feedback loop similar to the "Matthew Effect": operations that show early promise receive more gradient updates, which makes them even better, leading the search to prematurely converge on sub-optimal, "easy-to-train" designs. This unfair competition masks the true potential of superior architectures that simply require more time to mature.
Enforcing Strict Fairness
To combat this, researchers like Xiangxiang Chu and Bo Zhang introduced the concept of Strict Fairness. In their work on FairNAS, they argue that every architectural choice must receive an equal number of updates.
Instead of purely random sampling, they propose sampling without replacement. In each training step, every possible operation at a given layer is activated and updated exactly once. This ensures that the supernet acts as an unbiased proxy. By the time the search begins, the weights for a $5 \times 5$ convolution have been updated just as many times as the weights for a identity mapping, allowing for a "fair" head-to-head comparison. Xiangyu Zhang and the team behind Single Path One-Shot (SPOS) similarly found that uniform sampling is essential to decouple weight optimization from architecture selection, leading to much more stable rankings.
Balancing the Scales of Model Size
Fairness isn't just about which operation is chosen, but also the scale of the sub-network. In many search spaces, smaller models are mathematically more numerous than larger ones due to combinatorial effects.
Rhea Sanjay Sukthanker and Aaron Klein found that uniform sampling in Large Language Model (LLM) pruning often over-samples low-capacity models, leaving larger, more capable sub-networks under-trained (Compressing Large Language Models with Automated Sub-Network Search). To fix this, they use grid-based sampling to ensure that sub-networks across the entire parameter range—from tiny to massive—receive proportional training updates.
Similarly, Jordan Shipard demonstrated in Does Interference Exist When Training a Once-For-All Network? that the performance gap between large and small models in "Once-For-All" networks is often not due to weight interference, but simple sampling bias. By switching to a balanced Random Subnet Sampling (RSS) strategy, they achieved consistent performance across all model sizes without complex training schedules.
Structural Fairness and Normalization
Beyond sampling frequency, the physical structure of the supernet can introduce bias. In network width search, sub-networks often share the "first" $k$ channels. Xiu Su and colleagues observed in BCNet that this causes lower-indexed channels to be updated far more frequently than higher-indexed ones. Their solution, a bilaterally coupled network, ensures that every channel—regardless of its index—is trained equally.
Other researchers have focused on the role of Batch Normalization (BN). Jie Du identified that standard NAS search spaces unfairly favor operations that naturally include normalization, like convolutions, over identity paths. By implementing Fair BatchNorm (FBN-NAS)—placing normalization after every candidate operation—the search algorithm is forced to evaluate the operation's representative power rather than its learning stability (Training BatchNorm Only in Neural Architecture Search and Beyond).
Why It Works: Ranking Consistency
The ultimate goal of these fairness mechanisms is to improve ranking consistency, often measured by the Kendall-Tau ($\tau$) correlation. As Kaicheng Yu and Mathieu Salzmann emphasize, a supernet is only useful if its internal ranking of models matches the ground truth (How to Train Your Super-Net).
By ensuring that every operation, every channel, and every model scale is updated with equal frequency and capacity, we eliminate the "noise" of the training process. The supernet stops favoring the "fast learners" and starts identifying the truly "high achievers," turning weight-sharing NAS from a game of chance into a reliable tool for automated design.