A scalable algorithm for the optimization of neural network architectures

Massimiliano Lupo Pasini, Junqi Yin, Ying Wai Li, Markus EisenbachParallel Comput 2021 · DOI

TL;DR

This paper introduces GSNNA, a novel greedy algorithm for automatically designing neural network architectures. GSNNA efficiently finds architectures that perform as well as those found by other methods, but with significantly reduced computational cost and a more reliable search process, making it suitable for resource-constrained environments.

What Problem Does This Paper Address?

Designing the optimal architecture for a neural network (NN) is a critical but challenging task. The performance of an NN is highly sensitive to its structure, including the number of layers, neurons per layer, and connectivity. However, the sheer number of possible architectures grows exponentially, making exhaustive search infeasible. Existing hyperparameter optimization (HPO) methods, while effective, often struggle with scalability and can be computationally expensive, sometimes leading to unreliable estimates of performance. This paper addresses the need for a scalable and efficient HPO algorithm that can automatically discover high-performing NN architectures without requiring excessive computational resources or leading to uncertain results.

Methods

The core of this paper is the Greedy Search for Neural Network Architecture (GSNNA) algorithm. GSNNA takes a greedy approach to building NN architectures by incrementally adding hidden layers. At each step, it performs a stratified Random Search (RS), focusing the search on the hyperparameters of the newly added layer while keeping the configurations of previous layers fixed. This "recycling" of information from previously evaluated models allows GSNNA to exploit successful configurations and avoid exploring a vast, unmanageable search space.

Key aspects of GSNNA:

  • Greedy Layer Addition: Starts with a simple architecture (e.g., linear regression) and iteratively adds one hidden layer at a time.
  • Stratified Random Search: Within each iteration, RS is applied to determine the optimal number of neurons and other hyperparameters for the current hidden layer.
  • Information Recycling: The best-performing architecture from the previous iteration (with its layer configurations) is used as a starting point for the next iteration, freezing those configurations.
  • Scalability: The stratified RS and greedy approach significantly reduce the dimensionality of the hyperparameter search space at each step, leading to improved scalability.
  • Computational Efficiency: By avoiding the training of overly complex architectures that are later discarded, GSNNA aims for a lower computational cost per iteration compared to methods like Bayesian Optimization (BO) and Tree-Parzen Estimator (TPE).

Datasets

The authors evaluated GSNNA on five standard benchmark datasets:

  • Eggbox: An artificially constructed regression dataset.
  • Graduate Admission: A regression dataset predicting student admission chances.
  • Computer Hardware: A regression dataset relating hardware properties to CPU performance.
  • Phishing Websites: A classification dataset for identifying phishing websites.
  • CIFAR-10: A widely used image classification dataset.

MLP models were used for the first four datasets, while CNN models were employed for CIFAR-10.

Key Insights and Ideas

  • Greedy Layer Addition is Key: The paper argues that a greedy approach to adding layers, rather than treating the number of layers as just another hyperparameter to be optimized simultaneously with all others, is crucial for efficiency and scalability.
  • Stratified RS Reduces Uncertainty: By focusing the RS on the new layer's hyperparameters and reusing previous configurations, GSNNA leads to more reliable performance estimates with narrower confidence intervals compared to BO and TPE. This is attributed to reducing the dimensionality of the search space.
  • Computational Cost Reduction: GSNNA's iterative, greedy nature avoids the computationally expensive practice of training and discarding very large, complex models that can occur in other HPO methods.
  • Scalability Through Dimensionality Reduction: The method's design inherently tackles the "curse of dimensionality" in hyperparameter search by progressively exploring the architecture space.

Key Results

  • Performance: GSNNA achieved comparable or better predictive performance (measured by R² for regression and F1-score for classification) than BO and TPE across the tested datasets.
  • Scalability: GSNNA demonstrated significantly better weak scaling. As the number of concurrent model evaluations increased, GSNNA's computational time-to-solution increased much more gradually than BO and TPE, which saw substantial increases.
  • Reliability: GSNNA's confidence intervals for performance were narrower, indicating more stable and reliable results.
  • Efficiency: For a given number of concurrent model evaluations, GSNNA often achieved better performance or comparable performance in less time, especially at higher numbers of evaluations. For example, on the Eggbox dataset, GSNNA achieved better accuracy with narrower confidence bands as concurrent evaluations increased, while BO and TPE did not show this trend.

Limitations and Future Work

  • Dataset Size: While GSNNA shows promise, its application to extremely large datasets (like ImageNet) might still be computationally intensive due to the time required for individual model evaluations. The authors suggest that further research is needed to optimize for such scenarios.
  • Architecture Types: The current work focuses on MLPs and CNNs. Future work aims to extend GSNNA to other architectures like ResNets, RNNs, and LSTMs.
  • Uncertainty Quantification: The authors plan to conduct a more in-depth uncertainty quantification analysis to understand the sensitivity of hyperparameter inference to search space dimensionality and the number of concurrent evaluations.
  • Customized Metrics: Exploring the use of customized metrics beyond standard accuracy or F1-score for HPO is also a future direction.

Summary updated

BibTeX
@article{pasini2021scalable,
  author = {Massimiliano Lupo Pasini and Junqi Yin and Ying Wai Li and Markus Eisenbach},
  title = {{A scalable algorithm for the optimization of neural network architectures}},
  journal = {Parallel Comput.},
  year = {2021},
  doi = {10.1016/j.parco.2021.102788},
}
Figure