Neural Architecture Search

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, Tinghuai WangAutomated Machine Learning 2019 · DOI · DBLP

Neural Architecture Search (NAS) represents a significant shift in deep learning, moving from manual "architecture engineering" to automated model design. While deep learning successfully automated feature engineering, the design of the neural architectures themselves remained a manual, expert-driven process. This survey explores how NAS addresses the challenges of vast search spaces and high computational costs to produce state-of-the-art models.

The NAS Framework

The authors organize the NAS landscape into a three-part framework: the search space, the search strategy, and the performance estimation strategy.

The three-part framework of Neural Architecture Search, illustrating the interaction between the search space, search strategy, and performance estimation.
  1. Search Space: Defines the set of possible architectures. This has evolved from simple chain-structured networks to complex, multi-branch "cell-based" spaces.
  2. Search Strategy: Determines how to explore the search space, utilizing methods like Reinforcement Learning (RL), Evolutionary Algorithms, and Gradient-Based optimization (e.g., DARTS).
  3. Performance Estimation Strategy: Focuses on evaluating candidate architectures efficiently without full training, often through weight sharing (One-Shot NAS) or network morphisms.

Search Space Evolution

Early NAS research focused on chain-structured networks where layers were stacked linearly. Modern approaches often search for a small "cell" or motif (like a residual block) that is then stacked repeatedly to form the final network.

Comparison between simple chain-structured search spaces (left) and modern multi-branch cell-based search spaces (right).

This cell-based approach drastically reduces the search space and allows architectures discovered on small datasets, such as CIFAR-10, to be successfully transferred to larger datasets like ImageNet.

Key Insights and Results

  • Efficiency Gains: The shift from early RL methods to "One-Shot" weight-sharing methods has reduced search time from thousands of GPU days to just a few, with a reported 7x speedup in search time while maintaining performance.
  • State-of-the-Art Performance: NAS-generated models have achieved results surpassing many manual designs, reaching ~3.5% error on CIFAR-10 and competitive top-1 error rates on ImageNet (e.g., 20.3%).
  • The Search Space Bias: While NAS methods outperform random search, the margin is sometimes slim. This suggests that the design of the search space itself—often incorporating human-designed blocks—is as critical as the algorithm used to traverse it.

Future Directions

Despite its success, NAS faces several challenges. Most research is currently concentrated on image classification, necessitating a move toward other domains like generative models and semantic segmentation. Furthermore, there is a growing need for "Multi-Objective NAS" that optimizes for hardware constraints like latency and power consumption alongside accuracy. Finally, the field requires standardized benchmarks (like NAS-Bench-101) to ensure fair comparisons between emerging search strategies, moving beyond simple accuracy metrics to understand the "why" behind architectural success.

Summary updated

BibTeX
@inproceedings{elsken2019neural,
  author = {Thomas Elsken and Jan Hendrik Metzen and Frank Hutter and Tinghuai Wang},
  title = {{Neural Architecture Search}},
  booktitle = {Automated Machine Learning},
  year = {2019},
  doi = {10.1007/978-3-030-05318-5\_3},
}
Figure