Understanding Architectures Learnt by Cell-based Neural Architecture Search

Yao Shu, Wei Wang, Shaofeng Cai

Abstract

Neural architecture search (NAS) has become a cornerstone for automating model design, yet the structural properties of the resulting architectures remain poorly understood. This paper reveals that popular NAS algorithms (e.g., DARTS, ENAS) exhibit a systematic bias toward "wide and shallow" cell structures. While these architectures offer smoother loss landscapes and faster early-stage convergence, they do not necessarily provide superior generalization. Our study highlights a critical misalignment between search-phase trainability and final performance, suggesting a need for NAS evaluation metrics that prioritize true generalization over optimization speed.

The Problem: The NAS "Black Box"

Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of high-performance models that often surpass human-engineered counterparts. However, the "cells"—the fundamental building blocks discovered by these algorithms—are frequently treated as black boxes. Despite their success, there is a lack of clarity regarding why specific topological patterns are consistently selected and whether these patterns align with the ultimate goal of machine learning: generalization.

The Structural Bias: Wide and Shallow

Through a multi-stage investigation of popular cell-based NAS algorithms, including DARTS, ENAS, and SNAS, a clear pattern emerges. These algorithms consistently converge on architectures where most intermediate nodes are directly connected to input nodes. These cells are characterized as "wide" (featuring many parallel operations) and "shallow" (having few sequential steps).

Comparison of NAS-discovered cells and their variants, showing the transition from wide/shallow (a) to narrow/deep (e).

As shown in Figure 1, the original DARTS cell (a) is significantly wider and shallower than its variants. This structural preference is not accidental; it is a direct consequence of how NAS algorithms evaluate candidate architectures during the search phase. The researchers define "width" as the number of nodes connected to inputs and "depth" as the longest path through the cell.

The Optimization Trap

NAS algorithms typically rely on early-stage performance metrics, such as weight-sharing or evaluation after only a few epochs, to rank candidates. Wide and shallow models possess "benign" and smooth loss landscapes, which facilitate faster and more stable convergence early in the training process.

Optimization analysis using 2D loss contour plots and 3D gradient variance surface plots confirms that as cells become deeper and narrower, their loss landscapes become increasingly "agitated" and steep. This leads to higher gradient variance, making the optimization process noisier and slower. Consequently, deeper architectures—which might ultimately perform better—are often discarded prematurely because they appear less promising during the initial search phase.

Generalization vs. Trainability

The core insight of this research is the "generalization gap." The easiest model to train is not necessarily the best model for the task. In head-to-head tests on benchmarks like CIFAR-10, CIFAR-100, and Tiny-ImageNet-200, the original DARTS cell showed the fastest initial drop in test loss. However, some "sub-optimal" random variants actually outperformed it when trained to full convergence. For example, variant $C_7$ for DARTS and $C_4$ for ENAS achieved higher final test accuracy than the architectures selected by the algorithms.

Mathematically, the authors derived bounds for Lipschitz smoothness and gradient variance, proving that wider, shallower architectures are inherently easier to optimize. The Lipschitz constant for a narrow cell scales with the eigenvalues of the weight matrices, explaining the instability and slow convergence of deep/narrow cells during the search phase.

Conclusion and Future Directions

Current NAS algorithms are biased toward trainability rather than generalizability. This study suggests that the next frontier for NAS research lies in developing evaluation metrics that can estimate an architecture's final potential without the computational burden of full training. Moving beyond simple convergence speed will allow NAS to unlock more complex, deeper architectures that may offer superior performance on challenging real-world tasks.

Summary updated

Abstract

Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectures. In this paper, we first reveal that existing NAS algorithms (e.g., DARTS, ENAS) tend to favor architectures with wide and shallow cell structures. These favorable architectures consistently achieve fast convergence and are consequently selected by NAS algorithms. Our empirical and theoretical study further confirms that their fast convergence derives from their smooth loss landscape and accurate gradient information. Nonetheless, these architectures may not necessarily lead to better generalization performance compared with other candidate architectures in the same search space, and therefore further improvement is possible by revising existing NAS algorithms.
BibTeX
@inproceedings{shu2020understanding,
  author = {Yao Shu and Wei Wang and Shaofeng Cai},
  title = {{Understanding Architectures Learnt by Cell-based Neural Architecture Search}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=uIXBBehkUX},
}
Figure