Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates

Yingxia ShaoarXiv 2018 · arXiv

TL;DR

This paper introduces HOIST, a novel method for accelerating hyperparameter optimization (HPO) of deep neural networks (DNNs). HOIST leverages intermediate evaluation data from early stopping techniques, which are typically discarded, to build an ensemble of surrogate models, leading to significantly faster convergence and better performance compared to state-of-the-art methods.

What Problem Does This Paper Address?

Optimizing the hyperparameters of deep neural networks (DNNs) is crucial for achieving state-of-the-art performance, but it's a notoriously challenging and computationally expensive task. Traditional Bayesian Optimization (BO) methods, while effective, require a significant number of complete model evaluations to build accurate surrogate models of the performance landscape. Each complete evaluation can take hours or even days, severely limiting the number of configurations that can be explored within a practical timeframe. This "insufficient evaluation data" problem hinders the widespread adoption of BO for DNNs. Existing methods that incorporate early stopping (like Hyperband) accelerate HPO by allowing more configurations to be evaluated, but they often don't fully utilize the valuable intermediate performance data generated during these early stops.

Methods

HOIST tackles the "insufficient evaluation data" problem by proposing a new approach that utilizes intermediate evaluation data, rather than just discarding it. The core of HOIST lies in its ensemble surrogate modeling strategy:

  • Multiple Basic Surrogates: Instead of training a single surrogate model on all available data, HOIST trains multiple basic surrogate models (e.g., using SMAC, a random forest-based BO method) on different subsets of the mixed evaluation data. These subsets correspond to data collected at different stages of early stopping (e.g., data from early stops with fewer epochs, data from later stops with more epochs, and data from complete runs).
  • Ensemble Surrogate with Weighted Bagging: These multiple basic surrogates are then combined using weighted bagging to form a single, more accurate ensemble surrogate model. Each basic surrogate is assigned a weight, indicating its contribution to the ensemble's prediction.
  • Weight Vector Learning: A novel method is introduced to learn these weights. The weights are determined by the correlation between each basic surrogate's predictions and the true performance observed on the complete evaluation data. This ensures that surrogates that better approximate the objective function receive higher weights.
  • Integration with Early Stopping: HOIST integrates with early stopping methods like Hyperband. Instead of random sampling for Hyperband's inner loop, HOIST uses its ensemble surrogate to select promising configurations, further improving efficiency.

Datasets

The paper evaluates HOIST on several standard deep learning benchmarks:

  • MNIST: Used for tuning FCNet (Feed Forward Neural Network) and VAE (Variational Autoencoder).
  • CIFAR-10: Used for tuning CNNs (Convolutional Neural Networks).
  • IMDB: Used for tuning RNNs (Recurrent Neural Networks).

The experiments involve optimizing hyperparameters for these models, with varying numbers of hyperparameters (from 4 to 25) and maximum training epochs.

Key Insights and Ideas

  • Intermediate Data is Valuable: The paper's central insight is that intermediate evaluation data from early stopping, often considered noisy or incomplete, contains significant information about the performance landscape and can be effectively leveraged to accelerate BO.
  • Ensembling for Robustness: Combining multiple surrogates, each trained on data from different stages of early stopping, creates a more robust and accurate representation of the objective function than any single surrogate could achieve.
  • Correlation as a Weighting Mechanism: Using the correlation coefficient between surrogate predictions and actual performance on complete runs is an effective way to dynamically assign weights to ensemble members, prioritizing more accurate surrogates.
  • Synergy between Early Stopping and BO: HOIST demonstrates a powerful synergy by using early stopping to generate more data and then using that data to build better BO surrogates, creating a virtuous cycle of efficiency.

Key Results

HOIST consistently outperforms state-of-the-art methods across various DNN architectures and datasets:

  • Significant Speedups: HOIST achieves speedups of 3 to 6 fold in reaching near-optimal configurations compared to methods like Vanilla BO, Hyperband, HB-LCNet, and BOHB. For example, it reaches a similar validation error for FCNet on MNIST in 0.75 hours, while other methods take over 5 hours.
  • Improved Final Performance: HOIST consistently finds better hyperparameter configurations, achieving the best test performance on FCNet, CNN, RNN, and VAE tasks. For instance, it achieves a test error of 7.41% on FCNet, outperforming others.
  • Effective in High-Dimensional Spaces: HOIST demonstrates strong performance even in high-dimensional hyperparameter spaces (e.g., 25 hyperparameters for CNNs on CIFAR-10), where traditional BO methods struggle.
  • Handles Noisy Intermediate Data: Even when intermediate evaluation data quality is poor (as in the RNN on IMDB experiment), HOIST can still efficiently gather useful information and eventually outperform baselines.

Limitations and Future Work

  • Limitations:
    • The paper notes that for RNNs on IMDB, the intermediate evaluation data was of poor quality, leading to a slower initial convergence for HOIST. This suggests that the effectiveness of HOIST might depend on the quality and informativeness of the intermediate data.
  • The assumption of independence between basic surrogates ($f_i$ and $f_j$) is made for simplicity in calculating the ensemble variance, which might not always hold true.
  • Future Work:
    • The authors suggest exploring embedding HOIST into parallel and distributed computing environments to further accelerate HPO.
  • They also propose investigating the use of transfer learning techniques to combine basic surrogates, potentially improving generalization and efficiency.
  • The paper frames HOIST as a general ensemble framework, suggesting its applicability to other BO methods and early stopping techniques.

Summary updated

Abstract

The performance of deep neural networks crucially depends on good hyperparameter configurations. Bayesian optimization is a powerful framework for optimizing the hyperparameters of DNNs. These methods need sufficient evaluation data to approximate and minimize the validation error function of hyperparameters. However, the expensive evaluation cost of DNNs leads to very few evaluation data within a limited time, which greatly reduces the efficiency of Bayesian optimization. Besides, the previous researches focus on using the complete evaluation data to conduct Bayesian optimization, and ignore the intermediate evaluation data generated by early stopping methods. To alleviate the insufficient evaluation data problem, we propose a fast hyperparameter optimization method, HOIST, that utilizes both the complete and intermediate evaluation data to accelerate the hyperparameter optimization of DNNs. Specifically, we train multiple basic surrogates to gather information from the mixed evaluation data, and then combine all basic surrogates using weighted bagging to provide an accurate ensemble surrogate. Our empirical studies show that HOIST outperforms the state-of-the-art approaches on a wide range of DNNs, including feed forward neural networks, convolutional neural networks, recurrent neural networks, and variational autoencoder.
BibTeX
@misc{shao2018fast,
  author = {Yingxia Shao},
  title = {{Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates}},
  year = {2018},
  eprint = {1811.02319},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.02319},
}
Figure