Bayesian Optimization (BO) is the standard tool for "expensive" black-box problems—tasks where evaluating a single point might take hours of simulation or days of lab work. By building a probabilistic surrogate model (usually a Gaussian Process) of the objective function, BO intelligently balances exploring new regions with exploiting known peaks.
However, traditional BO hits a "wall" as the number of input variables increases. While it excels in 5 or 10 dimensions, performance often degrades significantly beyond 20 dimensions, a phenomenon frequently cited as the curse of dimensionality. As the search space volume grows exponentially, the surrogate model requires a prohibitive amount of data to remain accurate, and the acquisition function—the internal "guide" that picks the next point—becomes nearly impossible to optimize.
To scale BO to the hundreds or thousands of dimensions required for modern machine learning and engineering, researchers have moved away from treating the search space as a uniform "black box" and toward methods that exploit hidden structure.
Exploiting Low-Dimensional Structure
A recurring insight in high-dimensional optimization is that while a problem may have 100 variables, the objective function often only changes significantly along a few of them. Bo Chen and Andreas Krause explored this through variable selection, identifying which dimensions actually "matter" to the Gaussian Process.
Another approach is to assume the function is additive. Instead of modeling one massive 100-dimensional function, tools like Dragonfly, developed by Kirthevasan Kandasamy and Willie Neiswanger, decompose the objective into a sum of lower-dimensional functions. This reduces the data requirement from exponential to linear relative to the number of dimensions, allowing BO to handle significantly more complex engineering tasks.
Latent Space Optimization
When the input space is not only high-dimensional but also structured (like the pixels of an image or the parameters of a robot controller), researchers often map the high-dimensional space into a compact, low-dimensional latent space.
Rika Antonova and Akshara Rai demonstrated this in robotics with SVAE-DC. By using a Variational Autoencoder (VAE) to compress complex simulation data, they could perform BO in a small, manageable latent space. This allows robots to learn walking or manipulation tasks in just a dozen trials, even when the underlying controller has dozens of degrees of freedom. Similarly, for discrete sequences like proteins, Miguel González Duque notes that latent-space methods are essential for navigating the astronomical number of possible combinations.
Beyond Gaussian Processes
The computational cost of Gaussian Processes (GPs) scales cubically with the number of data points, which becomes a bottleneck in high-dimensional settings where many evaluations are needed. To solve this, researchers are swapping GPs for more scalable surrogate models.
- Bayesian Neural Networks: The PHOENICS algorithm, introduced by Florian Häse and Christoph Kreisbeck, uses neural networks to estimate the objective landscape. This provides a "universal" optimizer that maintains efficiency across hundreds of parameters.
- Random Forests: Toolkits like OpenBox leverage Random Forests, which handle high-dimensional and categorical inputs more robustly than standard GPs.
- Elastic Kernels: Santu Rana and Sunil Gupta proposed the Elastic Gaussian Process, which dynamically adjusts its "length-scale." This prevents the optimizer from getting stuck in the "flat terrain" that often characterizes high-dimensional acquisition functions.
Scaling to Thousands of Dimensions
For the most extreme cases—such as adversarial attacks on images or coordinating thousands of agents—even standard high-dimensional BO techniques can falter.
In adversarial ML, Sijia Liu and colleagues developed ZO-Min-Max, a zeroth-order method that bypasses the surrogate model entirely to estimate gradients directly. In multi-agent systems, Juliusz Krzysztof Ziomek and Ilija Bogunovic introduced Mean-Field BO. By optimizing the distribution of agent actions rather than each individual action, they achieved a complexity that is independent of the number of agents, effectively scaling BO to tens of thousands of dimensions.
Practical Bottlenecks
Despite these algorithmic advances, high-dimensional BO remains a practical challenge. As highlighted by the poli benchmark, many sophisticated algorithms suffer from memory exhaustion or excessive runtimes as dimensionality grows. In many real-world scenarios, simpler heuristics or randomized search strategies still serve as formidable baselines, suggesting that the gap between theoretical scalability and practical utility is a primary frontier for future research.