SuperNet in Neural Architecture Search: A Taxonomic Survey

Stephen Cha, Taehyeon Kim, Hayeon Lee, Se-Young Yun

Abstract

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.

Introduction: The Rise of the Supernet

Designing high-performance neural networks manually is a slow, expert-intensive process. Neural Architecture Search (NAS) was invented to automate this, but early methods were computationally ruinous, requiring thousands of GPU hours.

The Supernet emerged as a solution: a single, massive network that contains all possible architectures (subnets) within its weights. By training this one "master" network and allowing subnets to inherit its weights, researchers can theoretically find the best model for any device in a fraction of the time. However, Supernets face a significant hurdle known as poor rank correlation: the performance of a subnet when it is part of the Supernet often does not match its performance when trained from scratch. This discrepancy makes it difficult to identify which architecture is truly optimal. This survey categorizes the techniques used to mitigate this instability and improve search efficiency.

Taxonomy of Supernet Optimization

The survey proposes a two-pillar taxonomy for Supernet optimization, dividing methods into spatial and temporal categories.

Hierarchy of Supernet Optimization methods categorized by spatial and temporal dimensions.

1. Spatial Optimization

Spatial optimization focuses on the structural design and parameterization of the network.

  • Architecture Optimization: This is divided into Micro Optimization (designing small "cells" or building blocks to be repeated) and Macro Optimization (optimizing high-level connectivity and the combination of predefined blocks).
  • Parameter Optimization: This addresses the mechanics of weight sharing.
  • Local Weight-Sharing: Weights are shared only within specific sub-regions or hierarchies of the Supernet.
  • Global Weight-Sharing: Weights are shared across the entire search space. A common technique here is In-place Knowledge Distillation, where the largest possible subnet acts as a teacher for smaller ones to ensure consistency.

2. Temporal Optimization

Temporal optimization focuses on the strategy for sampling and training subnets over time.

  • Decoupled Optimization (One-Shot): The Supernet is trained first, and the search happens afterward. This includes Static Sampling (uniform random selection) and Dynamic Sampling (using greedy or attentive strategies to focus on "promising" subnets).
  • Coupled Optimization (Differentiable): The architecture and the weights are optimized simultaneously using gradient descent (e.g., DARTS). While fast, this approach is often prone to instability and convergence issues.

Methodological Landscape

The following table maps various state-of-the-art NAS methods to the proposed taxonomy, illustrating the diversity of approaches in the field.

A comprehensive mapping of existing NAS literature to the proposed taxonomy.

Key Insights and Performance Gains

Research into Supernet optimization has yielded several critical insights:

  • The Gradient Conflict Problem: A primary cause of poor rank correlation is "gradient conflict," where different subnets attempt to move shared weights in opposite directions during training.
  • "Train Once, Sample Many": The ultimate goal is a Supernet that allows for "direct deployment." If optimized correctly, one can select a subnet that fits specific hardware constraints (e.g., mobile latency) and use it immediately without retraining.
  • Efficiency Gains: Supernet-based NAS has reduced search costs from thousands of GPU days to less than 10 GPU hours in some differentiable cases.
  • Hardware Versatility: Methods like Once-for-All (OFA) demonstrate that a single trained Supernet can provide specialized models for diverse hardware (CPUs, GPUs, IoT devices) that outperform manually designed models like MobileNetV3.
  • Transformer Integration: Supernet optimization is now being successfully applied to Vision Transformers (ViTs), reducing their massive parameter counts while maintaining competitive accuracy.

Challenges and Future Directions

Despite significant progress, several challenges remain:

  • The Rank Correlation Gap: Perfectly predicting a subnet's standalone performance from its Supernet performance remains an open challenge.
  • Human Bias: Most NAS search spaces are still heavily constrained by human-designed "macro" structures. Future research may move toward more "evolutionary" searches that reconstruct algorithms from basic mathematical operations.
  • Stability vs. Speed: There is a persistent trade-off between the speed of coupled (differentiable) optimization and the stability of decoupled (one-shot) optimization. Finding a robust middle ground is a key area for future work.

Summary updated

Abstract

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.
BibTeX
@misc{cha2022supernet,
  author = {Stephen Cha and Taehyeon Kim and Hayeon Lee and Se-Young Yun},
  title = {{SuperNet in Neural Architecture Search: A Taxonomic Survey}},
  year = {2022},
  eprint = {2204.03916},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.03916},
}
Figure