Neural Architecture Search (NAS) has long been a "rich get richer" field. To find the best AI model, you traditionally needed thousands of GPU hours to train every candidate from scratch. To solve this, researchers developed "One-Shot" NAS, which uses a single massive "supernet" that shares weights across all possible architectures in a search space. While fast, this approach suffers from a fatal flaw: co-adaptation. When millions of different models share the same weights, the gradient updates for one architecture often interfere with another, leading to inaccurate performance estimates.
The papers in this cluster propose a middle ground: Few-Shot NAS. By partitioning the search space into specialized sub-regions, these methods mitigate interference while maintaining the efficiency of weight-sharing.
The Problem of Weight Co-adaptation
In a standard one-shot supernet, a single set of weights must represent a vast diversity of architectural paths. As Yiyang Zhao, Linnan Wang, and their colleagues point out in Few-Shot Neural Architecture Search, this creates a "ranking problem." The performance of a sub-network within the supernet often fails to correlate with its true performance when trained from scratch. This happens because the shared weights are forced to become "jacks of all trades," unable to specialize for any specific configuration.
Partitioning the Search Space
The core intuition behind Few-Shot NAS is to divide the massive search space into multiple disjoint regions, each represented by its own dedicated sub-supernet.
- Structural Partitioning: Zhao et al. demonstrate that using a small number of specialized sub-networks reduces the number of architectural paths each weight set must represent. This minimizes the "interference" between competing operations on the same edge.
- Capacity-Based Splitting: In the context of distilling large language models, Dongkuan Xu and Subhabrata Mukherjee introduce AutoDistil. They partition the space based on model scale, preventing performance-sapping interference between tiny student models and larger ones.
- Complexity Metrics: Youngmin Oh and Bumsub Ham suggest a "zero-cost" splitting criterion in Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions. By grouping architectures with similar numbers of ReLUs, they ensure that the supernets are trained on structurally similar models, further reducing gradient conflicts.
Why It Works: Specialization and Correlation
By moving from one global supernet to a few specialized ones, the search process gains two major advantages:
- Reduced Gradient Conflict: Weights within a sub-space only need to adapt to a subset of possible operations. This specialization allows the weights to better reflect the true potential of the architectures in that region.
- Higher Ranking Correlation: Because the "noise" from dissimilar architectures is removed, the relative ranking of models within a sub-supernet becomes much more reliable. As Zhao et al. show, even a minimal increase in computational overhead (training 5-10 sub-supernets instead of one) leads to a massive jump in the accuracy of performance predictions.
Beyond Architecture: Hyperparameters and Consensus
This partitioning logic extends beyond just neural layers. Xin Fang and Feng Cheng apply similar principles to high-dimensional hyperparameter optimization in BOASF, treating disjoint sub-spaces as independent arms in a multi-armed bandit framework. Even in blockchain research, Qing Lu and Yiyu Shi propose using these partitioned search spaces as a "Proof-of-Neural-Architecture" consensus mechanism, allowing miners to collaborate on different regions of the design space simultaneously.