Neural Architecture Search (NAS) represents a significant shift in deep learning, moving from manual "architecture engineering" to automated model design. While deep learning successfully automated feature engineering, the design of the neural architectures themselves remained a manual, expert-driven process. This survey explores how NAS addresses the challenges of vast search spaces and high computational costs to produce state-of-the-art models.
The NAS Framework
The authors organize the NAS landscape into a three-part framework: the search space, the search strategy, and the performance estimation strategy.
- Search Space: Defines the set of possible architectures. This has evolved from simple chain-structured networks to complex, multi-branch "cell-based" spaces.
- Search Strategy: Determines how to explore the search space, utilizing methods like Reinforcement Learning (RL), Evolutionary Algorithms, and Gradient-Based optimization (e.g., DARTS).
- Performance Estimation Strategy: Focuses on evaluating candidate architectures efficiently without full training, often through weight sharing (One-Shot NAS) or network morphisms.
Search Space Evolution
Early NAS research focused on chain-structured networks where layers were stacked linearly. Modern approaches often search for a small "cell" or motif (like a residual block) that is then stacked repeatedly to form the final network.
This cell-based approach drastically reduces the search space and allows architectures discovered on small datasets, such as CIFAR-10, to be successfully transferred to larger datasets like ImageNet.
Key Insights and Results
- Efficiency Gains: The shift from early RL methods to "One-Shot" weight-sharing methods has reduced search time from thousands of GPU days to just a few, with a reported 7x speedup in search time while maintaining performance.
- State-of-the-Art Performance: NAS-generated models have achieved results surpassing many manual designs, reaching ~3.5% error on CIFAR-10 and competitive top-1 error rates on ImageNet (e.g., 20.3%).
- The Search Space Bias: While NAS methods outperform random search, the margin is sometimes slim. This suggests that the design of the search space itself—often incorporating human-designed blocks—is as critical as the algorithm used to traverse it.
Future Directions
Despite its success, NAS faces several challenges. Most research is currently concentrated on image classification, necessitating a move toward other domains like generative models and semantic segmentation. Furthermore, there is a growing need for "Multi-Objective NAS" that optimizes for hardware constraints like latency and power consumption alongside accuracy. Finally, the field requires standardized benchmarks (like NAS-Bench-101) to ensure fair comparisons between emerging search strategies, moving beyond simple accuracy metrics to understand the "why" behind architectural success.
Summary updated