Sample Efficient Optimization of Complex Black Box Systems

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 146 papers48 conceptsfrozen 2026-10-04

In many scientific and engineering domains, the primary bottleneck to progress is the cost of evaluation. Whether it is a high-fidelity fluid-structure interaction simulation that takes days to run Christopher Earls, a physical experiment to synthesize a new polymer fiber Cheng Li, or a clinical trial for radiation therapy Qingying Wang, we cannot afford to "guess and check" thousands of times.

To solve this, researchers have converged on a shared strategy: Bayesian Optimization (BO). This approach treats the system as a "black box" and uses a probabilistic surrogate model to navigate the design space with extreme sample efficiency.

The Surrogate Model as a Cheap Proxy

The core intuition behind this research direction is that we can replace an expensive, slow evaluation with a fast, mathematical approximation. Most of the papers in this field, such as the work by Michael C. Burl and Esther Wang, utilize Gaussian Processes (GPs) as these surrogate models.

A GP doesn't just predict the outcome of a set of parameters; it predicts a probability distribution over possible outcomes. This means that for any point in the design space, the model provides both a predicted value and a measure of uncertainty (the "confidence interval"). This uncertainty is the secret sauce that allows the system to decide where to look next.

In Designing staggered platelet composite structure with Gaussian process regression based Bayesian optimization, Kundo Park and colleagues demonstrate this by optimizing bioinspired materials. Instead of 3D printing every possible geometry, they use the GP to "imagine" the performance of thousands of designs, only printing the ones the model is most curious or excited about.

The Acquisition Function: Balancing Exploration and Exploitation

If the surrogate model is the "map," the acquisition function is the "navigator." It uses the model’s predictions and uncertainty to pick the single most informative point to evaluate next.

The most common strategy is Expected Improvement (EI), which looks for points that are likely to beat the current best result. However, researchers are constantly refining this logic:

  • Dirk Husmeier and Umberto Noè proposed Scaled Expected Improvement (ScaledEI) to better account for uncertainty in the improvement itself, making the search more robust in noisy environments like medical diagnostics.
  • In the realm of Large Language Models (LLMs), Andrew Bell and Matthew Barker use advanced functions like $qLogNEHVI$ to handle multiple objectives—like balancing cost, latency, and safety—simultaneously.

This mechanism allows systems to find global optima in a fraction of the time. For example, Derssie Mebratu showed that BO could tune TensorFlow's CPU performance in just a few dozen iterations, a task that would normally take weeks of manual effort.

Real-world engineering is rarely a simple search for a maximum; it is usually a search for a maximum within limits. Jingyi Wang and Eric B. Chin applied constrained Bayesian Optimization to mechanical design, where parts must not exceed specific pressure limits. By incorporating these constraints directly into the acquisition function, the optimizer learns to avoid "illegal" regions of the design space without needing to be told where they are beforehand.

Similarly, Alaleh Ahmadianshalchi and Janardhan Rao Doppa developed PAC-MOO, a framework that handles massive design spaces where most configurations are infeasible. By using information-theoretic approaches, they can find high-performing, valid designs faster than traditional evolutionary algorithms.

From Physics to Software: A Universal Tool

The power of this research direction lies in its universality. Because it treats the underlying system as a black box, the same mathematical framework can be applied to vastly different problems:

As summarized by David Eriksson in his analysis of the NeurIPS Black-Box Optimization Challenge, Bayesian Optimization is fundamentally superior to random search because it learns from every single evaluation. In a world where evaluations are expensive, learning is the only way to scale.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sample_efficient_optimization_of_complex_60748db0,
  title = {Sample Efficient Optimization of Complex Black Box Systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-efficient-optimization-of-complex-black-box-systems/rdr_60748db0e4314d209ab489226e6be86f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sample_efficient_optimization_of_complex_60748db0_v1,
  title = {Sample Efficient Optimization of Complex Black Box Systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-efficient-optimization-of-complex-black-box-systems/rdr_60748db0e4314d209ab489226e6be86f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}