bbotk: Black-Box Optimization Toolkit

Marc Becker, Jakob Richter, Michel Lang, Bernd Bischl, Martin BinderOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

Features highly configurable search spaces via the 'paradox' package and optimizes every user-defined objective function. The package includes several optimization algorithms e.g. Random Search, Iterated Racing, Bayesian Optimization (in 'mlr3mbo') and Hyperband (in 'mlr3hyperband'). bbotk is the base package of 'mlr3tuning', 'mlr3fselect' and 'miesmuschel'.

Introduction

In many scientific and engineering fields, researchers encounter "black-box" functions—systems where internal mechanics are unknown or too complex to model analytically. Optimizing these functions, such as tuning hyperparameters for a neural network or finding the optimal shape for an airfoil, is notoriously difficult.

Historically, this field has been hindered by fragmented tools with inconsistent interfaces, making it difficult to switch between different optimization algorithms. Furthermore, managing evaluation histories, handling multi-objective trade-offs, and defining complex stopping conditions often requires significant custom code. bbotk (Black-Box Optimization Toolkit) addresses these challenges by providing a standardized, extensible, and object-oriented framework for the R programming language.

Methodology and Architecture

The toolkit is built on the R6 object-oriented system, ensuring a modular architecture where components can be mixed and matched. Key elements include:

  • Search Spaces (paradox): Leverages the paradox package to define complex parameter sets, including hierarchical dependencies and various data types (integers, doubles, categories).
  • Optimization Instances: Users define an OptimInstance which encapsulates the objective function, the search space, and the termination criteria. It supports both single-criterion and multi-criteria (Pareto) optimization.
  • Batch and Asynchronous Optimization: A standout feature is the support for different execution modes. While Batch optimization handles sequential groups of evaluations, the Asynchronous mode (utilizing the rush package and a Redis backend) allows for high-performance parallel evaluations across distributed workers.
  • Extensible Algorithms: The package includes built-in optimizers like Random Search, Grid Search, CMA-ES, and Iterated Racing (irace). It also provides a wrapper for the nloptr library, offering access to a wide array of non-linear optimization routines through a single interface.
  • Terminators: A dedicated class system for stopping optimization based on evaluation counts, clock time, performance levels, or stagnation.

For benchmarking and demonstration, the package utilizes standard mathematical functions such as the Branin function, a classic two-dimensional optimization benchmark.

Key Insights

Separation of Concerns

By decoupling the Objective (what to optimize), the Optimizer (how to search), and the Terminator (when to stop), bbotk allows researchers to experiment with different strategies by changing only a single line of code.

The Archive as a First-Class Citizen

Unlike many tools that only return the final "best" point, bbotk maintains a detailed Archive. This stores every evaluation, including timestamps and metadata, which is essential for reproducibility, multi-objective analysis, and understanding search behavior.

The Archive component in bbotk serves as the primary data storage for all evaluation results.

Modern Hardware Utilization

The integration of asynchronous optimization recognizes the shift toward distributed computing. It allows the system to propose new points for evaluation without waiting for all current workers to finish, maximizing hardware efficiency.

Ecosystem Integration and Results

bbotk serves as the foundational layer for several high-level packages in the mlr3 ecosystem:

  • mlr3tuning: For hyperparameter tuning.
  • mlr3fselect: For feature selection.
  • miesmuschel: For evolutionary algorithms.

The toolkit successfully provides a unified API for a diverse range of algorithms and natively handles multi-criteria optimization, allowing users to find and visualize Pareto fronts where trade-offs between competing objectives (e.g., accuracy vs. speed) are analyzed.

Limitations and Future Work

Currently, some local search algorithms do not automatically apply search space transformations, requiring manual handling within objective functions. Additionally, advanced optimizers like CMA-ES may encounter stability issues in extremely low-dimensional search spaces (fewer than three dimensions). Future development continues toward tighter integration with distributed backends and expanding the library of plug-and-play optimization algorithms.

Summary updated

Abstract

Features highly configurable search spaces via the 'paradox' package and optimizes every user-defined objective function. The package includes several optimization algorithms e.g. Random Search, Iterated Racing, Bayesian Optimization (in 'mlr3mbo') and Hyperband (in 'mlr3hyperband'). bbotk is the base package of 'mlr3tuning', 'mlr3fselect' and 'miesmuschel'.
BibTeX
@inproceedings{becker2020bbotk,
  author = {Marc Becker and Jakob Richter and Michel Lang and Bernd Bischl and Martin Binder},
  title = {{bbotk: Black-Box Optimization Toolkit}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.32614/cran.package.bbotk},
}
Figure