Understanding Architectures Learnt by Cell-based Neural Architecture Search

Yao Shu, Wei Wang, Shaofeng Cai

Neural Architecture Search (NAS) has been incredibly successful at automatically designing high-performing neural networks, yet the "why" behind its structural choices often remains a black box. This paper pulls back the curtain on these algorithms, revealing that popular NAS methods have a specific structural preference that might actually be a blind spot.

Existing cell-based NAS algorithms (like DARTS and ENAS) are heavily biased toward "wide and shallow" architectures. They favor these structures because they converge faster during the search phase. While these designs offer smoother loss landscapes and more stable gradients, they are not always the best for final generalization, suggesting that current NAS methods prioritize optimization speed over ultimate performance.

The "Wide and Shallow" Bias

Designing neural networks manually is notoriously difficult, which is why NAS was invented to automate the process. Typically, NAS searches for a "cell" (a small block of layers) that is then repeated to build a full network. However, most research has focused on making the search faster or more accurate, rather than analyzing the resulting architectures.

The authors noticed a recurring pattern: architectures found by different NAS algorithms look remarkably similar.

Architectures generated by popular NAS algorithms, showing a consistent preference for wide and shallow structures.

This observation led to three core questions:

  1. Is there a consistent structural pattern in NAS-generated cells?
  2. Why do NAS algorithms favor this pattern?
  3. Does this pattern actually lead to the best possible models?

Methodology

The researchers approached these questions through a combination of empirical observation and rigorous mathematical analysis:

  • Structural Metrics: They formally defined "width" (how many nodes connect directly to the input) and "depth" (the longest path through the cell).
  • Comparative Training: They compared popular NAS cells (from DARTS, ENAS, NASNet, etc.) against thousands of randomly connected variants with different widths and depths.
  • Landscape Visualization: They used loss contour mapping and 3D gradient variance surfaces to "see" how difficult it is to optimize different architectures.
  • Theoretical Proofs: They derived bounds for Lipschitz smoothness and gradient variance based on cell topology to prove why certain structures are easier to train.

The study utilized three standard benchmarks for image classification: CIFAR-10, CIFAR-100, and Tiny-ImageNet-200.

Key Insights

  • The "Premature Evaluation" Trap: Most NAS algorithms evaluate candidate architectures after only a few epochs of training to save time. This creates a massive selection bias. The algorithm isn't necessarily picking the best architecture; it's picking the fastest learner.
  • Wide and Shallow = Smooth Sailing: The authors found that wide and shallow cells create a "benign" loss landscape. In contrast, deep and narrow cells have "agitated" landscapes with steep, narrow regions that are difficult for optimizers to navigate.
  • Gradient Stability: Theoretically and empirically, the paper shows that as cells get deeper and narrower, the variance of the gradients explodes. This makes training unstable, especially during the early stages of the search process.
  • Optimization $\neq$ Generalization: Just because a model is easy to train doesn't mean it will perform best on new data. The authors found that some "random" variants that were harder to train eventually surpassed the "popular" NAS architectures in final test accuracy.

Results and The Generalization Gap

Across DARTS, ENAS, AmoebaNet, and SNAS, the "best" cells found were consistently the widest and shallowest in their respective search spaces. In head-to-head tests, the original DARTS cell converged significantly faster and more stably than its deeper, narrower variants.

Variants of the DARTS cell used to test the impact of width and depth on convergence and generalization.

However, this rapid convergence masks a generalization gap. When the authors "forced" architectures to be even wider and shallower than what NAS found, the test error often increased. Conversely, some variants that were harder to find during the search ultimately performed better. For example, on CIFAR-100, the original DARTS architecture had a 16.41% error, while a deeper variant achieved 16.15%.

Limitations and Future Work

The primary limitation of this study is its specific focus on cell-based NAS, where a single motif is repeated. It doesn't necessarily account for "macro" search spaces where the entire network topology is searched globally.

The authors suggest that future NAS research should move away from simple early-stage validation accuracy. Instead, the field needs new metrics that can estimate an architecture's true generalization potential without being fooled by how quickly it converges in the first few minutes of training. This could involve looking at the curvature of the loss landscape or the stability of gradients during the search itself.

Summary updated

Abstract

Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness has attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectures. In this paper, we first reveal that existing NAS algorithms (e.g., DARTS, ENAS) tend to favor architectures with wide and shallow cell structures. These favorable architectures consistently achieve fast convergence and are consequently selected by NAS algorithms. Our empirical and theoretical study further confirms that their fast convergence derives from their smooth loss landscape and accurate gradient information. Nonetheless, these architectures may not necessarily lead to better generalization performance compared with other candidate architectures in the same search space, and therefore further improvement is possible by revising existing NAS algorithms.
BibTeX
@inproceedings{shu2020understanding,
  author = {Yao Shu and Wei Wang and Shaofeng Cai},
  title = {{Understanding Architectures Learnt by Cell-based Neural Architecture Search}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=uIXBBehkUX},
}
Figure