BayesO: A Bayesian optimization framework in Python

Jungtaek Kim, Seungjin Choi, Jungtaek Kim 0001J. Open Source Softw 2023 · DOI · DBLPGitHub resource

BayesO Framework Logo

Introduction

The optimization of expensive black-box functions is a ubiquitous challenge in modern science and engineering. When the underlying objective function lacks a known analytical form or gradient information, and each evaluation incurs significant costs—whether in time, computational resources, or physical materials—traditional optimization strategies like grid search or random search become prohibitively inefficient.

BayesO is a lightweight, user-friendly Python framework designed to bridge the gap between complex Bayesian optimization (BO) theory and practical application. By leveraging probabilistic models to guide the search process, BayesO enables practitioners to identify global optima with a minimal number of function evaluations. This makes it an ideal tool for tasks ranging from hyperparameter tuning in machine learning to the design of complex chemical reactions.

The BayesO Framework

BayesO implements the standard iterative BO process while providing modular flexibility at each stage:

  1. Surrogate Modeling: The core of the BO process is the surrogate model, which provides a probabilistic approximation of the objective function. BayesO supports standard Gaussian Process (GP) regression but extends its utility by offering Student-$t$ process regression. This addition is particularly valuable for real-world applications where data may be noisy or contain significant outliers that would otherwise skew a standard GP model. For complex, non-smooth landscapes, the framework also includes Random Forest regression.
  2. Acquisition Functions: To determine the next point for evaluation, BayesO utilizes acquisition functions that mathematically represent the trade-off between exploration (sampling in regions of high uncertainty) and exploitation (sampling in regions where the model predicts a high value). The framework implements Expected Improvement (EI), Probability of Improvement (PI), and Upper Confidence Bound (GP-UCB).
  3. Query Point Selection: The optimization of these acquisition functions is handled through robust global and local solvers such as DIRECT, CMA-ES, and L-BFGS-B.

Software Engineering and Extensibility

Beyond its algorithmic capabilities, BayesO distinguishes itself through its commitment to software quality. The library features 100% unit test coverage, comprehensive type hinting, and strict adherence to Pythonic coding conventions. Licensed under the MIT license and easily installable via PyPI, it serves as an accessible entry point for researchers who need a reliable optimization loop without the overhead of deep-learning-centric frameworks like BoTorch. Its "scenario-based" wrappers further simplify the setup process, allowing users to easily incorporate existing experimental data into the optimization loop.

Experimental Demonstration

The framework's performance is illustrated using a synthetic 1D objective function defined as: $$f(x) = 2 \sin(x) + 2 \cos(2x) + 0.05x$$ Starting with only four initial observations, the framework iteratively refines its surrogate model over the search space $x \in [-5, 5]$. As shown in the following figure, the acquisition function $a(x)$ successfully guides the search toward the global maximum. By the ninth iteration, the model has accurately captured the function's landscape and identified the optimum with high precision.

Step-by-step Bayesian optimization iterations on a 1D synthetic function.

Conclusion

BayesO provides a robust, lightweight alternative for sample-efficient optimization. While it is primarily CPU-based and may face challenges in extremely high-dimensional spaces—a common limitation of Gaussian Process-based methods—its ease of use and software quality make it a dependable foundation for both practical optimization tasks and the development of new BO research. Future work will focus on expanding its algorithmic components and fostering community-driven improvements.

Summary updated

BibTeX
@article{kim2023bayeso,
  author = {Jungtaek Kim and Seungjin Choi and Jungtaek Kim and Seungjin Choi},
  title = {{BayesO: A Bayesian optimization framework in Python}},
  journal = {J. Open Source Softw.},
  year = {2023},
  doi = {10.21105/joss.05320},
}
Figure