Gradient Descent Effects on Differential Neural Architecture Search: A Survey

Santanu Santra, Jun-Wei Hsieh, Chi-Fang LinIEEE Access 2021 · OpenReview

Abstract

Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.

Introduction: The Quest for Automated Design

Manually designing a high-performing neural network is a "black art" that requires immense expertise and months of experimentation. Neural Architecture Search (NAS) aims to automate this process, replacing human intuition with algorithmic search. However, early NAS methods were prohibitively expensive, often requiring thousands of GPU hours to find a single optimal model.

The fundamental challenge lies in the search process itself. As shown in the general NAS pipeline, the system must navigate a vast search space to identify a candidate architecture, evaluate it, and iterate until an optimal configuration is found.

The general workflow of Neural Architecture Search (NAS), moving from search space definition to the final optimal architecture.

Initial approaches relied on Reinforcement Learning (RL) or Evolutionary Algorithms (EA), which treat the search like a discrete optimization problem—testing thousands of complete models to see which works best. This survey focuses on the shift toward Differentiable NAS (DNAS), which leverages gradient descent to make this process significantly more efficient.

Methodology: Differentiable NAS and Gradient Descent

The core innovation of Differentiable NAS is treating the architecture itself as a set of parameters that can be tuned using the same gradient descent methods used to train a model's weights. This transition from discrete "trial-and-error" to continuous optimization relies on two main techniques:

  1. Continuous Relaxation: Instead of making hard binary choices (e.g., choosing between a 3x3 Convolution or a Max Pool), DNAS represents the choice as a weighted sum of all possible operations. This makes the search space "smooth" and differentiable.
  2. Bi-level Optimization: The search becomes a two-part process. In one step, the algorithm updates the network's weights (learning the task); in the next, it updates the architecture parameters (refining the model structure).

The evolution of these search spaces has moved toward more specialized and efficient structures, with the DARTS (Differentiable Architecture Search) space currently being the most popular due to its balance of flexibility and performance.

The progression of NAS search spaces, highlighting the shift from early RL-based spaces to the currently popular DARTS space.

The survey also reviews standard gradient descent variants used in this process, such as SGD, Momentum, Adam, and AdaGrad, alongside "tricks of the trade" like Gradient Noise and Early Stopping that prevent the search from collapsing into poor-quality architectures.

Key Insights and Performance Benchmarks

The shift to gradient-based methods has yielded several critical insights:

  • The Efficiency Leap: Gradient-based methods (like DARTS) can find optimal architectures in roughly 1 GPU day, compared to the 2,000+ GPU days required by early RL-based methods.
  • The "Discretization Gap": A significant challenge is that while we search in a "fuzzy" continuous space, we must eventually select a "hard" discrete architecture. This "gap" often causes a sudden drop in performance that researchers are still trying to bridge.
  • The Skip-Connection Bias: Differentiable NAS tends to favor "easy" operations like skip-connections (identity mappings) because they allow gradients to flow easily during the search, even if they aren't the best choice for the final model.

In terms of results, modern DNAS methods like ProxylessNAS achieve error rates as low as 2.08% on CIFAR-10, rivaling or beating human-designed architectures. Furthermore, methods like GDAS can find robust architectures in just 4 GPU hours.

Challenges and Future Directions

Despite the rapid progress, several open problems remain:

  • Mathematical Foundations: Much of NAS currently relies on empirical results. There is a lack of rigorous mathematical proof explaining why certain discovered architectures outperform others.
  • Generalization: An architecture optimal for a small dataset like CIFAR-10 is often sub-optimal for larger ones like ImageNet. Creating "dataset-agnostic" search spaces remains a primary goal.
  • Unsupervised NAS: Most current NAS requires labeled data. The authors point toward UnNAS (Unsupervised NAS) as a promising frontier where architectures are discovered without human-annotated labels.
  • Hardware Awareness: Incorporating "hard" constraints like latency and FLOPs directly into the loss function allows the search to find models optimized for specific mobile or edge hardware.

By refining the application of gradient descent, NAS is moving from a computationally expensive luxury to a practical tool for everyday deep learning development.

Summary updated

Abstract

Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.
BibTeX
@inproceedings{santra2021gradient,
  author = {Santanu Santra and Jun-Wei Hsieh and Chi-Fang Lin},
  title = {{Gradient Descent Effects on Differential Neural Architecture Search: A Survey}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=6liCgKRHf7},
}
Figure