Abstract
Neural architecture search (NAS) has become a cornerstone for automating model design, yet the structural properties of the resulting architectures remain poorly understood. This paper reveals that popular NAS algorithms (e.g., DARTS, ENAS) exhibit a systematic bias toward "wide and shallow" cell structures. While these architectures offer smoother loss landscapes and faster early-stage convergence, they do not necessarily provide superior generalization. Our study highlights a critical misalignment between search-phase trainability and final performance, suggesting a need for NAS evaluation metrics that prioritize true generalization over optimization speed.
The Problem: The NAS "Black Box"
Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of high-performance models that often surpass human-engineered counterparts. However, the "cells"—the fundamental building blocks discovered by these algorithms—are frequently treated as black boxes. Despite their success, there is a lack of clarity regarding why specific topological patterns are consistently selected and whether these patterns align with the ultimate goal of machine learning: generalization.
The Structural Bias: Wide and Shallow
Through a multi-stage investigation of popular cell-based NAS algorithms, including DARTS, ENAS, and SNAS, a clear pattern emerges. These algorithms consistently converge on architectures where most intermediate nodes are directly connected to input nodes. These cells are characterized as "wide" (featuring many parallel operations) and "shallow" (having few sequential steps).
As shown in Figure 1, the original DARTS cell (a) is significantly wider and shallower than its variants. This structural preference is not accidental; it is a direct consequence of how NAS algorithms evaluate candidate architectures during the search phase. The researchers define "width" as the number of nodes connected to inputs and "depth" as the longest path through the cell.
The Optimization Trap
NAS algorithms typically rely on early-stage performance metrics, such as weight-sharing or evaluation after only a few epochs, to rank candidates. Wide and shallow models possess "benign" and smooth loss landscapes, which facilitate faster and more stable convergence early in the training process.
Optimization analysis using 2D loss contour plots and 3D gradient variance surface plots confirms that as cells become deeper and narrower, their loss landscapes become increasingly "agitated" and steep. This leads to higher gradient variance, making the optimization process noisier and slower. Consequently, deeper architectures—which might ultimately perform better—are often discarded prematurely because they appear less promising during the initial search phase.
Generalization vs. Trainability
The core insight of this research is the "generalization gap." The easiest model to train is not necessarily the best model for the task. In head-to-head tests on benchmarks like CIFAR-10, CIFAR-100, and Tiny-ImageNet-200, the original DARTS cell showed the fastest initial drop in test loss. However, some "sub-optimal" random variants actually outperformed it when trained to full convergence. For example, variant $C_7$ for DARTS and $C_4$ for ENAS achieved higher final test accuracy than the architectures selected by the algorithms.
Mathematically, the authors derived bounds for Lipschitz smoothness and gradient variance, proving that wider, shallower architectures are inherently easier to optimize. The Lipschitz constant for a narrow cell scales with the eigenvalues of the weight matrices, explaining the instability and slow convergence of deep/narrow cells during the search phase.
Conclusion and Future Directions
Current NAS algorithms are biased toward trainability rather than generalizability. This study suggests that the next frontier for NAS research lies in developing evaluation metrics that can estimate an architecture's final potential without the computational burden of full training. Moving beyond simple convergence speed will allow NAS to unlock more complex, deeper architectures that may offer superior performance on challenging real-world tasks.
Summary updated