Abstract
Expected Improvement (EI) is arguably the most popular acquisition function in Bayesian optimization and has found countless successful applications, but its performance is often exceeded by that of more recent methods. Notably, EI and its variants, including for the parallel and multi-objective settings, are challenging to optimize because their acquisition values vanish numerically in many regions. This difficulty generally increases as the number of observations, dimensionality of the search space, or the number of constraints grow, resulting in performance that is inconsistent across the literature and most often sub-optimal. Herein, we propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. We demonstrate that numerical pathologies manifest themselves in "classic" analytic EI, Expected Hypervolume Improvement (EHVI), as well as their constrained, noisy, and parallel variants, and propose corresponding reformulations that remedy these pathologies. Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions, highlighting the understated role of numerical optimization in the literature.
Introduction
Bayesian Optimization (BO) is the premier framework for optimizing expensive-to-evaluate black-box functions, such as hyperparameter tuning for deep learning or engineering design. The process relies on an acquisition function to determine the next point to sample. Expected Improvement (EI) has long been the industry standard due to its balance of exploration and exploitation. However, in recent years, EI has been frequently outperformed by more complex alternatives. This research demonstrates that EI's perceived weaknesses are often not due to its underlying logic, but rather to numerical pathologies that prevent effective optimization.
The Vanishing Gradient Pathology
The core issue identified is the "vanishing gradient" problem. In many regions of a search space, the probability of improvement is so minuscule that standard double-precision hardware rounds the value to exactly zero. When the acquisition value is zero, the gradient—the signal used by optimizers to find better points—also disappears.
As illustrated in Figure 1, the fraction of the search space where gradients vanish increases dramatically as the dimensionality ($d$) of the problem grows or as more training data points are added. This leaves standard optimizers "wandering in the dark," leading to the sub-optimal performance often attributed to the EI algorithm itself.
Methodology: The LogEI Family
To resolve these issues, the authors introduce LogEI, a family of acquisition functions that are mathematically equivalent or approximately equal to their original counterparts but are numerically stable.
- Analytic Log-Space: For standard EI, the authors derive a method to compute the logarithm of improvement directly. By utilizing a Laurent expansion, they ensure precise gradients even when the probability of improvement is as low as $10^{-100}$ or less.
- Smooth Approximations: For batch (qEI) and multi-objective (EHVI) settings, non-differentiable "hard" maximums are replaced with smooth approximations like softplus and p-norms.
- Fat-Tailed Non-linearities: To maintain a signal in large batches, the authors introduced "Fat Softplus" and "Fat Sigmoid" functions. These have polynomial rather than exponential decay, ensuring the optimizer receives a signal even from points far from the current best.
Empirical Results
LogEI was tested against a massive battery of benchmarks, including synthetic functions (Ackley, Levy), engineering design problems, and high-dimensional tasks like Rover trajectory planning.
The results, highlighted in Figure 6, show that LogEI continues to find improvements long after standard EI stalls. On the "Pressure Vessel" engineering problem, LogEI found a superior solution using three orders of magnitude fewer evaluations than previous genetic algorithms. Furthermore, LogEI matched or exceeded the performance of computationally expensive state-of-the-art methods like GIBBON and Joint Entropy Search (JES).
Conclusion
The success of LogEI suggests that many "state-of-the-art" acquisition functions developed recently may have only appeared superior because the baseline EI was poorly optimized. By addressing numerical stability, LogEI revitalizes a classic method, demonstrating that implementation details are just as critical as theoretical advancements in Bayesian Optimization.
Summary updated