Supervising the Multi-Fidelity Race of Hyperparameter Configurations

Martin Wistuba, Arlind Kadra, Josif Grabocka

TL;DR

DyHPO is a novel Bayesian Optimization framework that treats hyperparameter tuning as a "dynamic race" rather than a fixed tournament. By utilizing a deep kernel Gaussian Process that incorporates the actual shape of a model's learning curve, DyHPO makes smarter, real-time decisions about which configurations deserve more training budget, significantly outperforming industry standards like Hyperband and BOHB.

The Catch-22 of Hyperparameter Optimization

Hyperparameter Optimization (HPO) is the "black magic" of Deep Learning—choosing the right learning rate, dropout, or architecture can be the difference between a state-of-the-art model and a total failure. However, training deep networks is expensive. To save time, researchers use "multi-fidelity" (or gray-box) methods that train many configurations on small budgets (e.g., just a few epochs) and only promote the best-looking ones to full training.

The problem is that early performance is often a poor predictor of final success. For instance, a heavily regularized model might converge slowly and look "bad" at epoch 5, but eventually surpass a non-regularized model by epoch 50. Existing methods like Hyperband use rigid, pre-defined schedules (brackets) that often kill these "late bloomers" too early. The authors aim to solve this sub-optimal budget allocation by making the "race" dynamic.

Comparison of budget allocation and learning curves between Hyperband and DyHPO.

Introducing DyHPO: A Dynamic Race

The core of the paper is DyHPO (Dynamic HPO), which introduces two major innovations to address these rigid schedules:

  1. Deep Multi-Fidelity Surrogate: Traditional Bayesian Optimization uses Gaussian Processes (GP) with hand-engineered kernels. DyHPO uses a Deep Kernel. This neural network-based kernel takes three inputs: the hyperparameter configuration, the current budget, and—most importantly—the partial learning curve (the sequence of validation scores seen so far). It uses a 1D-Convolutional Neural Network (CNN) to extract features from this curve, allowing the model to "recognize" the dynamics of a promising vs. a failing training run.
  2. Multi-Fidelity Expected Improvement ($EI_{MF}$): The authors propose a new acquisition function. Instead of deciding which new configuration to start from scratch, $EI_{MF}$ looks at all currently running configurations and asks: "Which one of these is most likely to improve the current best score if I give it exactly one more unit of budget (e.g., one more epoch)?"

This transforms HPO into a dynamic race where the "referee" (the surrogate model) decides every single step which "runner" gets to keep going.

Key Insights

  • Learning Curves are Features: Most HPO methods treat a validation score at epoch 10 as just a single number. DyHPO treats the history from epoch 1 to 10 as a trajectory. This allows the model to distinguish between a configuration that is plateauing and one that is accelerating.
  • The "Late Bloomer" Problem: The paper proves that DyHPO is significantly better at identifying "good" configurations that performed poorly at low budgets. In their analysis, DyHPO promoted "good" configurations that were "bad" at early stages at a much higher rate than competitors.
  • Dynamic > Static: By moving away from the fixed "brackets" of Hyperband, DyHPO can pivot resources instantly. If a configuration shows a sudden burst of improvement, DyHPO can immediately prioritize it without waiting for a new bracket to start.

Experimental Results

The authors evaluated DyHPO across 50 different datasets spanning three modalities: LCBench (35 tabular datasets), TaskSet (12 NLP tasks), and NAS-Bench-201 (ImageNet, CIFAR-10, CIFAR-100). DyHPO was compared against seven strong baselines, including BOHB, DEHB, ASHA, and Dragonfly.

  • Superior Regret and Rank: Across almost all benchmarks, DyHPO achieved the lowest "regret" (the gap between the found solution and the global optimum) and the best average rank.
  • Efficiency: It reached high-performing configurations much faster than other methods. For example, on the LCBench benchmark, DyHPO outperformed all baselines significantly after using only 50% of the total search budget.
Convergence plot on ImageNet16-120 showing DyHPO achieving lower error faster than baselines.
  • Ablation Success: The authors tested a version of DyHPO without the CNN-based learning curve input. The version with the learning curve performed significantly better, proving that "watching the race" is the key to winning it.

Limitations and Future Work

While powerful, the authors note a few constraints:

  • Tabular Benchmark Gap: There is a lack of massive-scale tabular benchmarks for models like Transformers (GPT/BERT), so the performance on extremely large-scale HPO remains an open question.
  • Pause and Resume: The method relies on the ability to pause a training run and resume it later. While standard for neural networks, this might be technically challenging for certain legacy systems or specific hardware setups.
  • Overhead: For very small datasets where a model trains in milliseconds, the computational overhead of updating a Gaussian Process might actually be slower than just running a simple Random Search.

Ultimately, DyHPO represents a major step toward making HPO for Deep Learning practical, efficient, and "off-the-shelf" for researchers.

Summary updated

Abstract

Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
BibTeX
@inproceedings{wistuba2022supervising,
  author = {Martin Wistuba and Arlind Kadra and Josif Grabocka},
  title = {{Supervising the Multi-Fidelity Race of Hyperparameter Configurations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=0Fe7bAWmJr},
}
Figure