TL;DR
This paper introduces HOIST, a novel method for accelerating hyperparameter optimization (HPO) of deep neural networks (DNNs). HOIST leverages intermediate evaluation data from early stopping techniques, which are typically discarded, to build an ensemble of surrogate models, leading to significantly faster convergence and better performance compared to state-of-the-art methods.
What Problem Does This Paper Address?
Optimizing the hyperparameters of deep neural networks (DNNs) is crucial for achieving state-of-the-art performance, but it's a notoriously challenging and computationally expensive task. Traditional Bayesian Optimization (BO) methods, while effective, require a significant number of complete model evaluations to build accurate surrogate models of the performance landscape. Each complete evaluation can take hours or even days, severely limiting the number of configurations that can be explored within a practical timeframe. This "insufficient evaluation data" problem hinders the widespread adoption of BO for DNNs. Existing methods that incorporate early stopping (like Hyperband) accelerate HPO by allowing more configurations to be evaluated, but they often don't fully utilize the valuable intermediate performance data generated during these early stops.
Methods
HOIST tackles the "insufficient evaluation data" problem by proposing a new approach that utilizes intermediate evaluation data, rather than just discarding it. The core of HOIST lies in its ensemble surrogate modeling strategy:
- Multiple Basic Surrogates: Instead of training a single surrogate model on all available data, HOIST trains multiple basic surrogate models (e.g., using SMAC, a random forest-based BO method) on different subsets of the mixed evaluation data. These subsets correspond to data collected at different stages of early stopping (e.g., data from early stops with fewer epochs, data from later stops with more epochs, and data from complete runs).
- Ensemble Surrogate with Weighted Bagging: These multiple basic surrogates are then combined using weighted bagging to form a single, more accurate ensemble surrogate model. Each basic surrogate is assigned a weight, indicating its contribution to the ensemble's prediction.
- Weight Vector Learning: A novel method is introduced to learn these weights. The weights are determined by the correlation between each basic surrogate's predictions and the true performance observed on the complete evaluation data. This ensures that surrogates that better approximate the objective function receive higher weights.
- Integration with Early Stopping: HOIST integrates with early stopping methods like Hyperband. Instead of random sampling for Hyperband's inner loop, HOIST uses its ensemble surrogate to select promising configurations, further improving efficiency.
Datasets
The paper evaluates HOIST on several standard deep learning benchmarks:
- MNIST: Used for tuning FCNet (Feed Forward Neural Network) and VAE (Variational Autoencoder).
- CIFAR-10: Used for tuning CNNs (Convolutional Neural Networks).
- IMDB: Used for tuning RNNs (Recurrent Neural Networks).
The experiments involve optimizing hyperparameters for these models, with varying numbers of hyperparameters (from 4 to 25) and maximum training epochs.
Key Insights and Ideas
- Intermediate Data is Valuable: The paper's central insight is that intermediate evaluation data from early stopping, often considered noisy or incomplete, contains significant information about the performance landscape and can be effectively leveraged to accelerate BO.
- Ensembling for Robustness: Combining multiple surrogates, each trained on data from different stages of early stopping, creates a more robust and accurate representation of the objective function than any single surrogate could achieve.
- Correlation as a Weighting Mechanism: Using the correlation coefficient between surrogate predictions and actual performance on complete runs is an effective way to dynamically assign weights to ensemble members, prioritizing more accurate surrogates.
- Synergy between Early Stopping and BO: HOIST demonstrates a powerful synergy by using early stopping to generate more data and then using that data to build better BO surrogates, creating a virtuous cycle of efficiency.
Key Results
HOIST consistently outperforms state-of-the-art methods across various DNN architectures and datasets:
- Significant Speedups: HOIST achieves speedups of 3 to 6 fold in reaching near-optimal configurations compared to methods like Vanilla BO, Hyperband, HB-LCNet, and BOHB. For example, it reaches a similar validation error for FCNet on MNIST in 0.75 hours, while other methods take over 5 hours.
- Improved Final Performance: HOIST consistently finds better hyperparameter configurations, achieving the best test performance on FCNet, CNN, RNN, and VAE tasks. For instance, it achieves a test error of 7.41% on FCNet, outperforming others.
- Effective in High-Dimensional Spaces: HOIST demonstrates strong performance even in high-dimensional hyperparameter spaces (e.g., 25 hyperparameters for CNNs on CIFAR-10), where traditional BO methods struggle.
- Handles Noisy Intermediate Data: Even when intermediate evaluation data quality is poor (as in the RNN on IMDB experiment), HOIST can still efficiently gather useful information and eventually outperform baselines.
Limitations and Future Work
- Limitations:
- The paper notes that for RNNs on IMDB, the intermediate evaluation data was of poor quality, leading to a slower initial convergence for HOIST. This suggests that the effectiveness of HOIST might depend on the quality and informativeness of the intermediate data.
- The assumption of independence between basic surrogates ($f_i$ and $f_j$) is made for simplicity in calculating the ensemble variance, which might not always hold true.
- Future Work:
- The authors suggest exploring embedding HOIST into parallel and distributed computing environments to further accelerate HPO.
- They also propose investigating the use of transfer learning techniques to combine basic surrogates, potentially improving generalization and efficiency.
- The paper frames HOIST as a general ensemble framework, suggesting its applicability to other BO methods and early stopping techniques.
Summary updated