mlr3hyperband: Hyperband for 'mlr3'

Marc Becker, Sebastian Gruber, Jakob Richter, Julia Moosbauer, Bernd BischlOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

Successive Halving and Hyperband optimization algorithms are now available for the mlr3 ecosystem via the mlr3hyperband package. This implementation features improved scheduling and parallelizes the evaluation of configurations, offering both tuners for hyperparameter optimization in mlr3tuning and optimizers for general black-box optimization in bbotk. By leveraging multi-fidelity strategies, the package allows researchers to significantly accelerate the model tuning process by efficiently discarding poor configurations early in the evaluation cycle.

Introduction

Hyperparameter optimization (HPO) is frequently the most computationally intensive phase of the machine learning pipeline. Traditional methods, such as grid search or random search, are essentially "brute-force" approaches; they allocate identical resources—such as training epochs or data volume—to every configuration, regardless of its performance. This often results in massive computational waste on suboptimal models.

Multi-fidelity optimization addresses this by introducing a "budget" parameter. The goal is to evaluate configurations on a small budget initially, identifying and discarding "losers" early while promoting "winners" to higher budgets. mlr3hyperband provides a robust, parallelized implementation of these strategies specifically for R users working within the mlr3 framework.

Optimization Methods

The package implements three core optimization strategies designed to maximize efficiency:

  1. Successive Halving (SHA): Operating like a tournament, SHA starts with a large pool of configurations evaluated on a minimal budget. At each stage, only the top fraction (determined by the parameter $\eta$) survives to the next stage, where they are evaluated on an increased budget.
  2. Hyperband (HB): Recognizing that SHA can be sensitive to the initial trade-off between the number of configurations and the starting budget, Hyperband runs several "brackets" of SHA. Each bracket begins with a different balance of configurations and initial budgets, making the search more robust against poor initial settings.
  3. Asynchronous Successive Halving (ASHA): A critical feature of this implementation is support for asynchronous execution. In standard SHA, workers must wait for an entire stage to complete before promoting configurations. ASHA promotes configurations as soon as they reach the required threshold, maximizing hardware utilization and preventing "straggler" jobs from idling the cluster.

Configuration and Parameters

To utilize these algorithms effectively, users must define the budget range and the halving rate. These parameters are central to the performance of the tuner.

Core parameters for defining the Hyperband budget, including bounds and the halving factor.

As shown in the parameter definitions, the user must specify r_min (the lower bound of the budget) and r_max (the upper bound). The eta parameter determines the fraction of configurations that are promoted; for example, an eta of 3 means the top 1/3 of configurations move to the next stage with a budget increased by a factor of 3.

Key Insights and Ecosystem Integration

  • The Budget Requirement: The power of these algorithms relies on a meaningful "budget" parameter (e.g., nrounds in XGBoost or subsample in a pipeline). Through mlr3pipelines, users can apply these algorithms even to learners without a natural budget by using data subsampling as a proxy.
  • Custom Samplers: Unlike implementations that force uniform random sampling, mlr3hyperband allows users to utilize custom paradox samplers (e.g., Beta distributions) to bias the initial search space based on prior knowledge.
  • Parallelization: The implementation supports evaluating configurations of equal budget across different brackets in a single batch, significantly reducing wall-clock time in high-performance computing environments.

Limitations and Conclusion

While highly efficient, the effectiveness of Hyperband is dependent on the correlation between low-budget and high-budget performance. If a model's performance only stabilizes at very high budgets, SHA might discard it prematurely. Furthermore, the algorithm remains sensitive to the choice of $\eta$ and the budget range.

Despite these considerations, mlr3hyperband represents a significant step forward for the R community, providing a versatile tool for both machine learning tuning and general black-box function optimization. Its integration with the broader mlr3 ecosystem ensures that these advanced optimization strategies are accessible and scalable for complex research workflows.

Summary updated

Abstract

Successive Halving (Jamieson and Talwalkar (2016) &lt;<a href="https://doi.org/10.48550%2FarXiv.1502.07943" target="_top">doi:10.48550/arXiv.1502.07943</a>&gt;) and Hyperband (Li et al. 2018 &lt;<a href="https://doi.org/10.48550%2FarXiv.1603.06560" target="_top">doi:10.48550/arXiv.1603.06560</a>&gt;) optimization algorithm for the mlr3 ecosystem. The implementation in mlr3hyperband features improved scheduling and parallelizes the evaluation of configurations. The package includes tuners for hyperparameter optimization in mlr3tuning and optimizers for black-box optimization in bbotk.
BibTeX
@inproceedings{becker2020mlr,
  author = {Marc Becker and Sebastian Gruber and Jakob Richter and Julia Moosbauer and Bernd Bischl},
  title = {{mlr3hyperband: Hyperband for 'mlr3'}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.32614/cran.package.mlr3hyperband},
}
Figure