HyperJump: Accelerating HyperBand via Risk Modelling

Pedro Mendes, Maria Casimiro, Paolo Romano, David GarlanAAAI 2023 · DOI

Hyper-parameter optimization (HPO) is a critical yet computationally expensive component of modern machine learning workflows. While multi-fidelity methods like HyperBand (HB) have improved efficiency by evaluating configurations on small data subsets or fewer training epochs, they remain inherently "blind." HB allocates significant resources to configurations destined for rejection because it fails to leverage data from previous evaluations to predict future performance.

In their AAAI-23 paper, researchers from INESC-ID and Carnegie Mellon University introduce HyperJump (HJ). This method integrates model-based risk analysis into the HyperBand framework, allowing the optimizer to "jump" over unnecessary evaluation stages and achieve massive speedups without sacrificing robustness.

Methodology: Risk-Aware Jumping

HyperJump builds upon the robust foundation of HyperBand but adds a predictive "brain" using Gaussian Process (GP) surrogate models. The core innovation is the ability to skip intermediate budget stages entirely if the estimated risk is low.

The HyperJump mechanism compared to standard HyperBand, illustrating how risk thresholds allow configurations to skip stages.

Key components of the HJ framework include:

  • Expected Accuracy Reduction (EAR): A novel risk metric that estimates the potential loss in accuracy if the algorithm skips a stage and accidentally discards a high-quality configuration.
  • Relative EAR (rEAR): A normalized version of EAR used to decide if a jump is "safe." If the rEAR falls below a threshold (typically 10%), HJ skips the current stage and moves the most promising configurations directly to a higher budget.
  • Safest Set Selection: A lightweight heuristic that identifies the optimal subset of configurations to include in a jump to minimize aggregate risk.
  • Prioritization: HJ prioritizes configurations expected to provide the most information to the surrogate model, thereby reducing the risk of future jumps.

Experimental Evaluation

The authors evaluated HyperJump across several challenging benchmarks, including NATS-Bench (Neural Architecture Search on ImageNet-16-120 and CIFAR-100) and LIBSVM (SVM tuning on the Covertype dataset). The experiments compared HJ against state-of-the-art optimizers such as BOHB, ASHA, and Fabolas in both sequential and parallel (32-worker) settings.

Key Results and Insights

HyperJump consistently outperformed all baselines across diverse tasks and hardware configurations:

  • Massive Speedups: On NATS-Bench, HJ achieved 20x to 32x speedups compared to the best-performing baselines in sequential deployments. On the LIBSVM benchmark, it was approximately 7x faster than HB and BOHB.
  • Parallel Efficiency: In high-concurrency setups (32 workers), HJ maintained a significant lead, achieving up to 6x speedups over ASHA, an optimizer specifically designed for parallelization.
  • Ablation Study: Analysis revealed that the "jumping" logic is the single most important factor in HJ's performance. By skipping the low-fidelity "noise" of early stages, the optimizer reaches near-optimal configurations significantly faster than standard halving methods.
Comprehensive performance results across ImageNet, CIFAR-100, and SVM benchmarks, including parallel scaling and ablation studies.

Conclusion and Future Work

HyperJump demonstrates that model-based risk analysis can drastically accelerate multi-fidelity HPO. By quantifying the risk of discarding configurations, HJ effectively navigates the exploration-exploitation trade-off more efficiently than "blind" search methods.

While HJ is highly effective, its performance remains dependent on the surrogate model's ability to accurately represent the search space. Future research directions include adapting HJ's risk-modeling logic to fully asynchronous frameworks and exploring its sensitivity in extremely noisy or complex hyper-parameter landscapes.

Summary updated

BibTeX
@inproceedings{mendes2023hyperjump,
  author = {Pedro Mendes and Maria Casimiro and Paolo Romano and David Garlan},
  title = {{HyperJump: Accelerating HyperBand via Risk Modelling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26097},
}

Loading related content...

Figure