Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning
Lists
Only you see this
Abstract
Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods. However, the multi-budget search mechanisms of existing prior works can suffer from the poor correlation among the performances of hyperparameter configurations at different budgets. As a remedy, we introduce DyHPO, a method that learns to dynamically decide which configuration to try next, and for what budget. Our technique is a modification to the classical Bayesian optimization for a gray-box setup. Concretely, we propose a new surrogate for Gaussian Processes that embeds the learning curve dynamics and a new acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP, CNN/NAS, RNN).
BibTeX
@inproceedings{wistuba2022dynamic,
author = {Martin Wistuba and Arlind Kadra and Josif Grabocka},
title = {{Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning}},
booktitle = {Advances in Neural Information Processing Systems},
year = {2022},
url = {https://openreview.net/forum?id=0Fe7bAWmJr},
}Loading related content...