For years, the consensus in machine learning was that "vanilla" Bayesian Optimization (BO) — the combination of a Gaussian Process (GP) surrogate and an acquisition function like Expected Improvement (EI) — simply couldn't handle high-dimensional search spaces. Researchers pivoted toward increasingly complex architectures, dimensionality reduction, and specialized priors to make BO work in spaces with dozens or hundreds of variables.
However, a series of recent findings suggests that the "failure" of standard BO in high dimensions isn't a theoretical limitation, but a numerical one. The culprit is a phenomenon known as gradient vanishing, which effectively "blinds" the optimizer, making it appear as though the algorithm has stalled when it has actually just fallen off a numerical cliff.
The Numerical Cliff of High Dimensions
In Bayesian Optimization, we use a surrogate model (usually a GP) to predict the objective function and an acquisition function to decide where to sample next. Both components rely on gradients to work: we optimize the GP's hyperparameters to fit the data, and we optimize the acquisition function to find the next best point.
As Leonard Papenmeier, Matthias Poloczek, and Luigi Nardi argue in Understanding High-Dimensional Bayesian Optimization, high-dimensional BO often fails because these gradients hit zero. When the gradients vanish, the search process reverts to an uninformative random search. The algorithm isn't "stuck" in a local optimum; it's lost in a perfectly flat landscape where every direction looks the same to a computer using floating-point math.
Why Gaussian Processes "Flatline"
The most common surrogate model, the Gaussian Process, relies on a "length-scale" parameter that determines how far its influence extends from observed data points. In high dimensions, the distance between points grows rapidly (the "curse of dimensionality").
Zhitong Xu, Shandian Zhe, and their colleagues point out in Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization that if the initial length-scale is too small relative to these distances, the GP treats every data point as independent. The resulting model is a "flat-lined" response: it predicts the mean everywhere except at the exact coordinates of the data points. Because the model is flat, the gradients used to train it vanish, and the model never learns to be better.
The fix is surprisingly simple: by initializing length-scales with a $\sqrt{d}$ scaling (where $d$ is the number of dimensions), the GP maintains a "connection" between points, allowing gradients to flow and the model to learn effectively.
The Underflow of Expected Improvement
Even if the surrogate model is perfect, the acquisition function can still fail. Expected Improvement (EI) calculates the likelihood that a new point will beat the current best. In high dimensions or when far from the data, this probability can become infinitesimally small — say, $10^{-100}$.
Standard floating-point precision cannot represent such small numbers; it rounds them to zero. As Sebastian Ament, Samuel Daulton, and the team at Meta demonstrate in Unexpected Improvements to Expected Improvement for Bayesian Optimization, this numerical underflow creates "flat regions" in the search space. If the acquisition function is zero everywhere, the gradient-based optimizer used to find the next sample point stalls.
By implementing acquisition functions in a way that avoids this underflow — such as using log-space transformations — the authors show that "old reliable" methods like EI can actually outperform modern, complex alternatives.
Intuition: The Fog of Dimensionality
Imagine trying to find the peak of a mountain in a thick fog using only the slope of the ground beneath your feet.
- The GP Failure: If your "vision" (length-scale) is too short, you can't see the slope at all. You think you're on a flat plain because you can't see the next hill.
- The Acquisition Failure: Even if you have a map, if the slope is so gentle that your level can't measure it, you don't know which way to walk.
The recent research in this area suggests that we don't need a better map or a faster way to walk; we just need to clear the fog (better initialization) and use a more sensitive level (numerical stability). When these "plumbing" issues are fixed, standard Bayesian Optimization remains a formidable tool for even the most complex high-dimensional problems.