Abstract
Sparse Gaussian processes are a key component of high-throughput Bayesian optimisation (BO) loops; however, we show that existing methods for allocating their inducing points severely hamper optimisation performance. Standard strategies focus on global regression accuracy, spreading modeling resources across the entire search space. We introduce a new strategy that intelligently clusters "inducing points" in promising regions, allowing for high-precision optimisation without the cubic computational cost of traditional GPs.
The Scalability Bottleneck in Bayesian Optimisation
Bayesian Optimisation (BO) is a powerful framework for optimizing expensive, noisy "black-box" functions, such as tuning chemical reactions or hyperparameter optimization. Traditionally, BO relies on Gaussian Processes (GPs) to model the objective function. However, GPs suffer from a major scalability bottleneck: their computational cost grows cubically ($O(N^3)$) with the number of observations $N$.
To maintain efficiency in high-throughput scenarios, researchers utilize Sparse Gaussian Processes, which approximate the full dataset using a small set of $M$ "inducing points." The critical challenge is Inducing Point Allocation (IPA)—deciding where these points should be placed. Most existing IPA methods are designed for global regression, aiming to minimize uncertainty everywhere. In the context of BO, this is inefficient; we do not need a perfect model of sub-optimal regions. Instead, we require high fidelity near the global optimum.
Methodology: Quality-Diversity Decomposition
We propose a novel IPA strategy specifically tailored for BO. Our approach utilizes Determinantal Point Processes (DPPs), a mathematical framework for selecting subsets of points that are both high-quality and diverse.
The core innovation is the Quality-Diversity Decomposition. While standard methods focus primarily on diversity (repelling points to cover the space), we introduce a Quality Function $q(z)$. This function acts as a magnet, pulling inducing points toward regions likely to contain the optimum. We developed three variants:
- IMP-DPP (Improvement-based): Uses an "Expected Improvement" metric for single-objective optimization.
- AL-DPP (Active Learning): Focuses points near decision boundaries for classification tasks.
- HV-DPP (Hyper-volume): Targets the Pareto front in multi-objective optimization.
Experimental Evaluation
We evaluated these methods against standard baselines like K-means clustering and Conditional Variance Reduction (CVR) across synthetic benchmarks (e.g., Michalewicz, Shekel) and real-world simulations, including a 9D Heat Exchanger design problem.
Key Results
- High Precision: On complex functions like Michalewicz, IMP-DPP was the only sparse method to achieve the same precision as an "Exact GP."
- Computational Efficiency: In the 9D Radiator design task, our method achieved results comparable to an Exact GP with an order-of-magnitude reduction in wall-clock time.
- Constant Overhead: Unlike exact GPs, our sparse approach maintains a constant computational cost regardless of the number of evaluations, which is essential for long-running, high-throughput optimization loops.
Discussion and Future Work
The primary insight of this work is that inducing points should be treated as a "modeling budget" that can be "gerrymandered" to favor neighborhoods where the optimum is likely to reside. Spreading points too thin leads to "blurry" posterior means where the optimizer literally cannot "see" sharp peaks.
Future research will focus on improving the approximations for information-theoretic variants (like ENT-DPP) and extending these targeted allocation strategies to non-Euclidean search spaces, such as molecular graphs or discrete strings.
Summary updated