Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Martin Wistuba, Arlind Kadra, Josif Grabocka

Abstract

Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods. However, the multi-budget search mechanisms of existing prior works can suffer from the poor correlation among the performances of hyperparameter configurations at different budgets. As a remedy, we introduce DyHPO, a method that learns to dynamically decide which configuration to try next, and for what budget. Our technique is a modification to the classical Bayesian optimization for a gray-box setup. Concretely, we propose a new surrogate for Gaussian Processes that embeds the learning curve dynamics and a new acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP, CNN/NAS, RNN).
BibTeX
@inproceedings{wistuba2022dynamic,
  author = {Martin Wistuba and Arlind Kadra and Josif Grabocka},
  title = {{Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=0Fe7bAWmJr},
}

Loading related content...

Figure