Understanding Architectures Learnt by Cell-based Neural Architecture Search

Yao Shu, Wei Wang, Shaofeng Cai

TL;DR

This paper reveals that popular cell-based Neural Architecture Search (NAS) algorithms inherently favor wide and shallow cell structures. While these architectures optimize quickly due to smoother loss landscapes and lower gradient variance, they do not necessarily yield better final generalization performance than deeper, narrower alternatives.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of high-performing neural networks, often outperforming human experts. However, NAS algorithms are largely treated as black boxes.

The authors tackle a fundamental question: Do popular cell-based NAS algorithms (like DARTS, ENAS, and NASNet) share a common structural pattern, and if so, why do these algorithms prefer it? Understanding this bias is crucial for identifying flaws in current search strategies and designing more robust next-generation NAS frameworks.

Methods

To systematically analyze NAS-generated networks, the authors first introduce formal definitions for a cell's depth (the longest path from input to output nodes) and width (the number of parallel channels/connections directly linked to input nodes).

They evaluate popular NAS cells against two types of control baselines:

  1. Topological variants: Randomly connected cells designed to span a spectrum of widths and depths.
  2. Operational variants: Cells keeping the original NAS topology but utilizing randomized operations (e.g., swapping a convolution for a pooling layer).

To uncover the underlying optimization mechanics, the authors visualize the empirical loss landscapes using 2D loss contours and map gradient variance using 3D surface plots. They complement these empirical findings with a rigorous theoretical analysis, proving bounds on block-wise Lipschitz smoothness and gradient variance for extreme cell topologies (widest vs. narrowest).

Datasets

The architectures and their variants were trained and evaluated on three standard image classification benchmarks:

  • CIFAR-10
  • CIFAR-100
  • Tiny-ImageNet-200

Key Insights and Ideas

  • The "Wide and Shallow" Bias: SOTA cell-based NAS algorithms consistently converge on cells that are exceptionally wide and shallow, where the majority of intermediate nodes connect directly to the input nodes.
  • The Premature Evaluation Trap: Most NAS algorithms evaluate candidate architectures prematurely—either using weight-sharing approximations or training for only a few epochs. This creates a strong selection bias toward architectures that make rapid early progress.
  • Optimization $\neq$ Generalization: Wide and shallow cells possess a benign, near-convex loss landscape and lower gradient variance, which mathematically guarantees faster and more stable early convergence. However, this optimization advantage does not mean they generalize better at full convergence.

Key Results

  • Convergence Speed: The original wide/shallow NAS cells consistently exhibit faster and more stable early training curves compared to their narrower, deeper counterparts.
Topologies of DARTS cell and its random variants, showing decreasing width and increasing depth from left to right.
  • Loss Landscape Smoothness: Visualizations confirm that as cell width increases and depth decreases, the near-convex region around the optimum widens significantly. Deeper, narrower cells result in highly agitated, non-convex landscapes.
  • The Generalization Gap: When trained completely to convergence with full regularization, the NAS-selected architectures are frequently outperformed by randomly generated deeper/narrower variants. For instance, specific random variants achieved higher final test accuracy on CIFAR-10 than the architectures discovered by DARTS and ENAS.

Limitations and Future Work

The primary limitation highlighted is that current NAS evaluation metrics are flawed proxies for final performance because they over-index on early optimization speed. While wide and shallow structures serve as excellent architectural priors for stable training, blindly favoring them restricts the search space.

The authors suggest that future NAS research must move away from proxy-based premature performance indicators and focus on developing search policies that can efficiently estimate true generalization performance at full convergence.

Summary updated

Abstract

Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectures. In this paper, we first reveal that existing NAS algorithms (e.g., DARTS, ENAS) tend to favor architectures with wide and shallow cell structures. These favorable architectures consistently achieve fast convergence and are consequently selected by NAS algorithms. Our empirical and theoretical study further confirms that their fast convergence derives from their smooth loss landscape and accurate gradient information. Nonetheless, these architectures may not necessarily lead to better generalization performance compared with other candidate architectures in the same search space, and therefore further improvement is possible by revising existing NAS algorithms.
BibTeX
@inproceedings{shu2020understanding,
  author = {Yao Shu and Wei Wang and Shaofeng Cai},
  title = {{Understanding Architectures Learnt by Cell-based Neural Architecture Search}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=uIXBBehkUX},
}
Figure