Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Martin Wistuba, Arlind Kadra, Josif Grabocka

Abstract

Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).

Introduction

Training deep learning models is a resource-intensive process. To optimize performance, researchers utilize "gray-box" HPO methods that evaluate configurations on partial budgets (e.g., a few epochs) and terminate those that appear unpromising. However, a significant challenge persists: early-stage performance is often a poor predictor of final accuracy. Rigid, bracket-based methods like Hyperband may prematurely discard "slow-starting" models that would eventually achieve superior results.

To address this, we present DyHPO (Dynamic Hyperparameter Optimization), a framework that replaces static allocation schedules with a continuous, dynamic decision-making process.

Methodology: The Dynamic Race

DyHPO treats the optimization process as a "dynamic race" where configurations are evaluated step-by-step. The core innovation lies in its surrogate model and acquisition strategy.

Deep Multi-Fidelity Surrogate

Unlike standard Gaussian Processes, DyHPO employs a Deep Kernel GP. This model uses a neural network feature extractor to process the hyperparameter configuration ($x_i$), the current budget level ($j$), and the entire observed learning curve history ($Y_{i,j-1}$).

The DyHPO surrogate architecture uses 1D convolutions and global max pooling to embed learning curve history, enabling the model to recognize convergence patterns.

By utilizing 1D convolutions, the surrogate can "read" the shape of the learning curve, identifying whether a model is likely to plateau or continue improving.

Multi-Fidelity Expected Improvement (EIMF)

Decisions are guided by the EIMF acquisition function. At each step, DyHPO calculates which active configuration has the highest probability of improving the current best result if trained for exactly one more budget unit. This allows for a granular, adaptive allocation of resources.

Experimental Evaluation

We evaluated DyHPO against seven strong baselines (including BOHB, DEHB, and ASHA) across a massive benchmark suite of 50 datasets:

  • LCBench: 35 tabular datasets using MLP architectures.
  • TaskSet: 12 NLP tasks using Recurrent Neural Networks (RNNs).
  • NAS-Bench-201: Neural Architecture Search benchmarks on CIFAR and ImageNet.

Key Results

DyHPO consistently achieved lower regret—the difference between the found solution and the global optimum—across all benchmarks. Notably, it often reached optimal performance levels using significantly less budget than traditional methods.

Mean regret comparison on LCBench, TaskSet, and ImageNet16-120 benchmarks. DyHPO (gray line) consistently outperforms state-of-the-art baselines in both convergence speed and final performance.

Statistical tests (Friedman and Wilcoxon) confirmed that DyHPO’s performance gains are significant. Furthermore, wall-clock time experiments showed that the improved budget allocation efficiency far outweighs the computational cost of maintaining the Deep GP surrogate.

Conclusion and Future Work

DyHPO demonstrates that incorporating the full history of learning curves into a dynamic allocation strategy significantly improves HPO efficiency. While currently limited to parametric models that support "pause and resume" functionality, DyHPO sets a new standard for gray-box optimization. Future research will focus on extending this framework to multi-objective optimization and larger-scale architectures like Transformers.

Summary updated

Abstract

Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
BibTeX
@inproceedings{wistuba2022dynamic,
  author = {Martin Wistuba and Arlind Kadra and Josif Grabocka},
  title = {{Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=0Fe7bAWmJr},
}
Figure