Bayesian Optimization for Adaptive Experimental Design: A Review

Stewart Greenhill, Santu Rana, Sunil Gupta, Pratibha Vellanki, Svetha VenkateshIEEE Access 2020 · OpenReview

Introduction

Scientific and engineering breakthroughs—from discovering novel alloys to optimizing complex manufacturing processes—often rely on expensive and time-consuming experimental cycles. Traditionally, researchers have used Design of Experiments (DOE) methods, such as factorial designs or Latin hypercubes. These methods are static; they determine a sampling pattern before any measurements are taken.

Traditional static DOE methods, like this central composite design, determine sampling points before the experiment begins.

Because traditional DOE is fixed, it cannot adapt to interesting features or unexpected results that emerge during the experimental process. Bayesian Optimization (BO) offers a transformative alternative by modeling and optimizing these "black-box" functions adaptively, minimizing the total number of experiments required.

The Bayesian Optimization Framework

The core of BO is a two-part iterative loop:

  1. Surrogate Modeling: A Gaussian Process (GP) acts as a statistical surrogate, predicting outcomes across the design space and, crucially, quantifying the uncertainty of those predictions.
  2. Acquisition Function: A mathematical policy (e.g., Expected Improvement or Upper Confidence Bound) uses the surrogate's predictions to balance exploration (sampling uncertain areas) and exploitation (sampling promising areas) to suggest the next best experiment.
Bayesian Optimization adaptively samples the design space, focusing on areas of high interest or uncertainty as modeled by a surrogate.

Tackling Real-World Bottlenecks

While "vanilla" BO is powerful, real-world experimental design introduces several complexities that the authors survey:

  • High Dimensionality: Standard BO performance often degrades beyond 15-20 dimensions. Modern techniques use random embeddings or additive structures to optimize spaces with up to 50 dimensions.
  • Physical Constraints: Researchers often face "black-box" constraints where a setting's validity is unknown until the experiment fails. Methods like weighted Expected Improvement or Augmented Lagrangian approaches help navigate these boundaries.
  • Batch and Parallel Processing: To speed up discovery, BO can be adapted to suggest diverse batches of experiments to run in parallel using local penalization techniques, rather than waiting for single results.
  • Mixed-Variable Types: Real experiments often mix continuous, discrete, and categorical variables. Swapping GPs for Random Forests or using specialized kernels allows BO to handle these heterogeneous inputs.

Strategic Insights for Experimentalists

A key takeaway from the review is the power of Transfer Learning and Multi-Fidelity Optimization. Prior knowledge from past experiments or cheap, low-fidelity simulations can be embedded into the GP kernel. This allows the optimizer to "jump-start" the physical experiment, fusing massive amounts of cheap data with a few high-fidelity lab tests to reach the optimum faster.

However, a significant challenge remains: Composability. While solutions exist for individual problems (like batching or constraints), combining them is mathematically complex. The field currently lacks frameworks that allow researchers to easily "mix-and-match" these advanced capabilities without breaking the underlying statistical models.

Performance and Software Ecosystem

The review highlights that while GP-based BO is computationally intensive, modern GPU-accelerated libraries like GPflowOpt or MOE can achieve 100x speedups over single-threaded CPU implementations. The authors map out the software landscape, noting that libraries like GPyOpt and DiceOptim are currently leading the way in supporting advanced features like mixed factor types and parallel optimization.

Conclusion

Bayesian Optimization provides a robust, adaptive roadmap for modern experimental design. By moving beyond static sampling, it offers a path to more efficient discovery in science and engineering. Future research must focus on creating composable algorithms and providing stronger theoretical guarantees for these increasingly complex, multi-faceted optimization systems.

Summary updated

Abstract

Bayesian optimisation is a statistical method that efficiently models and optimises expensive “black-box” functions. This review considers the application of Bayesian optimisation to experimental design, in comparison to existing Design of Experiments (DOE) methods. Solutions are surveyed for a range of core issues in experimental design including: the incorporation of prior knowledge, high dimensional optimisation, constraints, batch evaluation, multiple objectives, multi-fidelity data, and mixed variable types.
BibTeX
@inproceedings{greenhill2020bayesian,
  author = {Stewart Greenhill and Santu Rana and Sunil Gupta and Pratibha Vellanki and Svetha Venkatesh},
  title = {{Bayesian Optimization for Adaptive Experimental Design: A Review}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=qBhtpYpWs7R},
}
Figure