Abstract
Multi-fidelity (gray-box) hyperparameter optimization (HPO) techniques have emerged as a promising direction for tuning Deep Learning methods. However, existing methods often suffer from sub-optimal budget allocation. We introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race. By employing a novel deep kernel for Gaussian Processes that embeds learning curve dynamics and an acquisition function incorporating multi-budget information, DyHPO demonstrates significant superiority over state-of-the-art methods across 50 diverse datasets spanning tabular, image, and NLP tasks.
The Challenge: Beyond Black-Box Optimization
Hyperparameter Optimization (HPO) is a critical yet computationally expensive component of modern Deep Learning. Traditional Bayesian Optimization (BO) treats training runs as "black boxes," requiring a model to finish training completely before its performance can be evaluated. This is often prohibitively slow for large-scale networks.
To address this, "gray-box" or multi-fidelity methods like Hyperband and BOHB have been developed. These methods use early stopping to discard poorly performing models based on fixed, "static" schedules or brackets. However, these static approaches face a significant hurdle: the poor rank correlation between early and late performance. A "late bloomer"—a model that starts slowly due to high regularization but eventually achieves superior accuracy—is often prematurely terminated by static methods.
Methods: The DyHPO Framework
DyHPO (Dynamic HPO) shifts the paradigm from static schedules to a dynamic, epoch-by-epoch allocation of training budgets. It treats HPO as a "dynamic race" where every configuration is re-evaluated at each step.
Deep Multi-Fidelity Surrogate
At the heart of DyHPO is a Deep Kernel Gaussian Process. Unlike standard GPs that use simple kernels, DyHPO utilizes a neural network feature extractor ($\phi$) to process three distinct inputs:
- Hyperparameter Configuration ($x_i$): The specific settings being tested.
- Current Budget ($j$): The number of epochs trained so far.
- Partial Learning Curve ($Y_{i,j-1}$): The sequence of validation scores observed up to the current point.
To effectively capture the temporal dynamics of training, the partial learning curve is processed through a 1D-Convolutional Neural Network (CNN). This allows the surrogate model to "recognize" patterns of convergence, distinguishing between a model that is failing and one that is simply converging steadily.
Multi-Fidelity Expected Improvement ($EI_{MF}$)
DyHPO employs a modified acquisition function, $EI_{MF}$. At each step of the race, the algorithm calculates the expected improvement of training a specific configuration for exactly one additional unit of budget (e.g., one more epoch). This fine-grained evaluation allows for a truly dynamic allocation of resources.
Key Insights: The Dynamic Race
The core philosophy of DyHPO is that the shape of the validation curve is a first-class feature. By treating the learning curve as a vital signal, DyHPO can recover "late bloomers" that other methods would discard.
As shown in the comparison above, static methods like Hyperband (left) operate in rigid brackets, often spending budget on many configurations that are eventually discarded. In contrast, DyHPO (right) focuses its computational budget heavily on the most promising configurations, allowing them to "race" ahead while quickly pausing those that show little promise of improvement.
Experimental Evaluation
The researchers conducted a large-scale evaluation of DyHPO across 50 different datasets and three modalities:
- LCBench: 35 tabular datasets using Multi-Layer Perceptron (MLP) architectures.
- TaskSet: 12 NLP tasks using Recurrent Neural Networks (RNNs).
- NAS-Bench-201: 3 image classification datasets (CIFAR-10, CIFAR-100, ImageNet-16-120) for Neural Architecture Search.
Results and Significance
DyHPO consistently outperformed seven strong baselines, including BOHB, DEHB, and ASHA, achieving lower "mean regret" (the difference between the found solution and the global optimum).
Statistical analysis using Friedman tests and Critical Difference diagrams confirmed that DyHPO’s performance lead is statistically significant across nearly all benchmarks. In terms of efficiency, DyHPO reached high-performing configurations significantly faster in wall-clock time, demonstrating its ability to pivot resources toward a new leader the moment the data supports it.
Limitations and Future Work
While powerful, DyHPO has certain constraints:
- Benchmark Reliance: The study primarily uses tabular benchmarks with pre-computed results, which may not fully capture the noise of live training environments.
- Pause/Resume Overhead: The strategy assumes that pausing and resuming training is efficient. While true for most frameworks, it may be a bottleneck in specific hardware setups.
- Computational Cost of the Surrogate: For very small models that train in seconds, the overhead of updating the Deep GP surrogate might outweigh the benefits of early stopping. In these niche cases, simpler methods like Random Search may remain competitive.
Future work involves extending DyHPO to handle even larger search spaces and exploring its application in real-time training of massive Transformer models.
Summary updated