Abstract
This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.
Introduction
The manual design of deep learning architectures is a labor-intensive process requiring significant domain expertise. Neural Architecture Search (NAS) emerged to automate this design, but early iterations—often based on reinforcement learning or evolutionary algorithms—were computationally prohibitive, requiring thousands of GPU days to find a single optimal model.
The introduction of the Supernet has fundamentally changed this landscape. A supernet is a single, over-parameterized network that contains every possible architecture in a defined search space as a "subnet." Through weight-sharing, subnets inherit parameters from the supernet, enabling a "train once, sample many" paradigm. This allows researchers to train one massive model and then instantly extract optimized sub-models for diverse hardware targets, from mobile CPUs to cloud GPUs.
However, the primary challenge in supernet optimization is the rank correlation problem. Often, a subnet's performance within the supernet does not accurately reflect its performance when trained independently (standalone). This survey explores the spatial and temporal optimization strategies developed to ensure that internal supernet rankings translate to real-world performance.
Taxonomy of Supernet Optimization
To organize the vast literature on this topic, the authors propose a taxonomy based on two primary pillars: spatial optimization and temporal optimization. Spatial optimization focuses on the structural definition and weight-sharing mechanisms, while temporal optimization addresses the scheduling and methodology of subnet selection during the training process.
Spatial Optimization
Spatial optimization defines what is being optimized within the network's structure and how its parameters are managed.
Architecture Optimization
This involves defining the granularity of the search space.
- Micro Optimization: Focuses on designing a single "cell" or block (e.g., a specific combination of convolutions and activations) that is repeated throughout the network. Methods like DARTS and ENAS are classic examples.
- Macro Optimization: Searches for the optimal global connectivity between pre-defined blocks, determining the overall depth and width of the network. Frameworks like DNA and BossNAS utilize this approach to find high-level structural efficiencies.
Parameter Optimization
This addresses the mechanics of weight-sharing.
- Local Weight-Sharing: Partitions the supernet into sub-regions to improve training stability and reduce interference between disparate subnets.
- Global Weight-Sharing: Shares weights across the entire search space. A key technique is In-place Knowledge Distillation, where the largest possible subnet (the "supernet" itself) acts as a teacher for smaller subnets, ensuring that all models in the space learn consistent and high-quality feature representations.
Temporal Optimization
Temporal optimization focuses on how and when subnets are selected and updated during the training lifecycle.
Decoupled Optimization (One-Shot NAS)
In decoupled optimization, the training of the supernet and the search for the optimal architecture are separate phases.
- Static Sampling: Subnets are picked randomly during training to ensure uniform coverage of the search space (e.g., SPOS).
- Dynamic Sampling: Uses greedy algorithms or "attentive sampling" to focus training on more promising or challenging architectures, improving the final rank correlation (e.g., AttentiveNAS).
Coupled Optimization (Differentiable NAS)
Coupled optimization performs training and searching simultaneously. By relaxing the discrete search space into continuous architecture parameters, these systems (like DARTS) can use gradient descent to "learn" the architecture itself alongside the network weights. While efficient, these methods can sometimes be unstable, collapsing into simple solutions like excessive skip-connections.
Literature and Applications
The diversity of supernet optimization methods reflects the broad range of challenges in the field. Figure 2 maps prominent NAS frameworks to the taxonomy and highlights their extensions into various application domains.
Beyond standard benchmarks like ImageNet and CIFAR-10, supernet optimization is increasingly applied to:
- Transformers: Optimizing head counts, embedding sizes, and layer depths in Vision Transformers (ViT) and NLP models.
- Specialized Domains: Tailoring models for satellite imagery, IoT devices, and domain-generalized tasks where data distributions may shift.
Key Insights and Future Directions
The shift toward supernet-based NAS has yielded dramatic efficiency gains, reducing search times from thousands of GPU days to under 10 hours in many cases. Models like Once-for-All (OFA) have demonstrated that it is possible to generate hardware-versatile models that outperform handcrafted state-of-the-art architectures like MobileNetV3 without additional training costs.
However, several open questions remain. There is a persistent theoretical gap in understanding why weight-sharing works as well as it does, and gradient conflicts between subnets remain a "silent killer" of performance. Future research is trending toward Meta-Learning—developing models that learn how to search—and incorporating latent space optimization to further bridge the gap between supernet training and real-world deployment.
Conclusion
Supernet optimization has transformed Neural Architecture Search from a niche, resource-heavy academic exercise into a practical tool for industrial-scale model deployment. By categorizing methods into spatial and temporal strategies, this survey provides a roadmap for navigating the complex landscape of modern automated machine learning.
Summary updated