Gradient-based Optimization of Neural Network Architecture

Will Grathwohl, Elliot Creager, Seyed Kamyar Seyed Ghasemipour, Richard S. ZemelWorkshops @ ICLR 2018 · OpenReview

Introduction

Designing an optimal neural network architecture is traditionally a manual, "black-box" process. Parameters such as the number of layers (depth) or the number of units per layer (width) are discrete, meaning it is impossible to directly compute the gradient of the loss with respect to them.

Historically, researchers have relied on methods like grid or random search, which are expensive and inefficient, or Bayesian Optimization, which treats the model as a black box and requires training many full models to completion.

To address these inefficiencies, this paper introduces Variational Model Optimization (VMO). VMO is a method designed to jointly optimize a neural network's architecture and its weights in a single training pass. By moving architecture search into the inner loop of training, the network can dynamically "learn" its own optimal structure—pruning or expanding as needed—via standard backpropagation.

Methods: Variational Model Optimization

The authors frame architecture search as a Variational Optimization problem. Instead of selecting one fixed architecture $A$, they define a distribution $q_\phi(A)$ over possible architectures and minimize the expected loss over this distribution.

To make this process compatible with backpropagation, the authors employ specific strategies for depth and width, alongside a continuous relaxation trick.

Depth and Width Optimization

  1. Depth Optimization: The authors construct a network with a maximum depth $D$. Every intermediate layer is equipped with its own "classification head." The final output is a weighted sum of these heads, with weights determined by a categorical distribution. During training, the model learns to shift probability mass onto the specific layer that offers the best trade-off between accuracy and complexity.
  2. Width Optimization: To optimize width, the authors utilize a masking approach. If a layer has a maximum of $K$ units, a binary mask is applied to "turn off" units beyond a certain index.
Depth and width optimization strategies

The Concrete Relaxation

Because categorical choices (like selecting a specific layer or masking a unit) are discrete, standard gradients cannot flow through them. To solve this, the authors use the Concrete (or Gumbel-Softmax) relaxation.

This technique replaces discrete samples with "soft" continuous approximations, making the entire architecture-selection process differentiable. A temperature parameter $t$ is used to "anneal" the distribution over time. As $t$ decreases, the soft approximations harden, forcing the model to eventually commit to a single discrete architecture by the end of training.

Annealing the temperature parameter in the Concrete relaxation

Key Insights and Results

The core conceptual shift in VMO is treating the architecture as a learnable parameter rather than a static "meta-setting." Unlike Reinforcement Learning-based Neural Architecture Search (NAS), which can take thousands of GPU hours, VMO searches for the architecture during the standard training time of a single model. By adding a complexity penalty $r(A)$ to the loss function, the model is incentivized to find the smallest architecture that maintains high performance.

The authors evaluated VMO using the Wide ResNet (WRN-28-10) base model on the CIFAR10 and CIFAR100 datasets:

  • Depth Optimization: On CIFAR10, the model successfully learned that it only needed 26 layers instead of the baseline 28. It achieved a slightly higher accuracy (96.01% vs 96.00%) while reducing the parameter count by approximately 8%.
  • Width Optimization: The model significantly reduced layer widths (down to ~76-81% of the original capacity). While this caused a minor drop in accuracy (95.22% vs 96.00% on CIFAR10), it yielded a vastly more compact model.
  • Joint Optimization: When optimizing depth and width simultaneously, the resulting models became even more compact, though the accuracy trade-off was more pronounced (e.g., 76.23% on CIFAR100 compared to the 80.75% baseline).

Limitations and Future Work

While VMO offers a fast, single-pass alternative to traditional NAS, it has limitations. The Concrete relaxation is a biased estimator, which the authors suspect is the primary reason for the slight performance degradation observed in the width and joint optimization tasks. Additionally, the initial "exploration" phase—where the model evaluates different architectures—can slow down the convergence of the network's weights.

Future work could expand this approach to more complex architectural choices, such as skip-connection patterns or kernel sizes. Furthermore, utilizing unbiased gradient estimators (like REBAR or RELAX) might improve final model performance, provided they can be effectively scaled to large convolutional networks.

Summary updated

Abstract

Neural networks can learn relevant features from data, but their predictive accuracy and propensity to overfit are sensitive to the values of the discrete hyperparameters that specify the network architecture (number of hidden layers, number of units per layer, etc.). Previous work optimized these hyperparmeters via grid search, random search, and black box optimization techniques such as Bayesian optimization. Bolstered by recent advances in gradient-based optimization of discrete stochastic objectives, we instead propose to directly model a distribution over possible architectures and use variational optimization to jointly optimize the network architecture and weights in one training pass. We discuss an implementation of this approach that estimates gradients via the Concrete relaxation, and show that it finds compact and accurate architectures for convolutional neural networks applied to the CIFAR10 and CIFAR100 datasets.
BibTeX
@inproceedings{grathwohl2018gradient,
  author = {Will Grathwohl and Elliot Creager and Seyed Kamyar Seyed Ghasemipour and Richard S. Zemel},
  title = {{Gradient-based Optimization of Neural Network Architecture}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
  url = {https://openreview.net/forum?id=C8_c3FD3i6},
}
Figure