Abstract
Sparse Gaussian Processes are a key component of high-throughput Bayesian Optimisation (BO) loops; however, we show that existing methods for allocating their inducing points severely hamper optimisation performance. By exploiting the quality-diversity decomposition of Determinantal Point Processes, we propose the first inducing point allocation strategy designed specifically for use in BO. Unlike existing methods which seek only to reduce global uncertainty in the objective function, our approach provides the local high-fidelity modelling of promising regions required for precise optimisation. More generally, we demonstrate that our proposed framework provides a flexible way to allocate modelling capacity in sparse models and so is suitable broad range of downstream sequential decision making tasks.
Introduction
Bayesian Optimization (BO) has emerged as a premier framework for optimizing expensive black-box functions, particularly in fields like experimental design and hyperparameter tuning. At its core, BO relies on a surrogate model—typically a Gaussian Process (GP)—to provide uncertainty estimates that guide the search for the global optimum. However, standard GPs face a significant "computational tax": their complexity scales cubically ($O(N^3)$) with the number of observations.
In modern "high-throughput" scenarios, where thousands of function evaluations are performed, this cubic scaling becomes a bottleneck. To maintain efficiency, researchers employ Sparse GPs, which approximate the full dataset using a small set of $M$ "inducing points." While Sparse GPs reduce the computational burden, their performance is heavily dependent on the placement of these inducing points. This paper addresses a critical flaw in current Inducing Point Allocation (IPA) strategies and proposes a task-specific alternative designed for optimization.
The Problem: Global Accuracy vs. Local Precision
Existing IPA strategies, such as K-means clustering or Conditional Variance Reduction (CVR), are designed for global regression accuracy. They aim to spread inducing points across the entire search space to minimize the overall approximation error. While this is ideal for general modeling, it is often counterproductive for Bayesian Optimization.
In BO, we do not require high accuracy in sub-optimal regions of the search space. Instead, we need high-fidelity modeling in regions where the global optimum is likely to reside. Standard IPA methods often "waste" their limited modeling capacity—what we might call a "pixel budget"—on regions that have no impact on the final optimization result. This lack of local resolution can lead to poor decision-making and a failure to converge on the true optimum.
Methodology: Quality-Diversity DPPs
To solve this, we introduce a novel framework for IPA that leverages the Quality-Diversity decomposition of Determinantal Point Processes (DPPs). This approach allows us to balance two competing needs: the need to explore the space (Diversity) and the need to model promising areas with high precision (Quality).
The Quality Function
We introduce a Quality Function $q(z)$ that assigns a score to different regions of the search space based on their relevance to the task. For single-objective optimization, we propose the Improvement-based DPP (IMP-DPP). Here, the quality function is derived from the "Expected Improvement" (EI) metric. By prioritizing points with high EI, the model is incentivized to place more inducing points in areas where the global minimum is likely to be found.
Greedy Selection
To select the inducing points, we use a greedy algorithm that maximizes the product of quality and diversity. This ensures that the resulting set of points provides a high-resolution "zoom-in" on the most interesting parts of the landscape while maintaining enough global awareness to prevent the optimizer from getting trapped in local minima.
Beyond Single-Objective BO
A key strength of this framework is its flexibility. By simply changing the quality function, the method generalizes to other tasks:
- Active Learning: The quality function can focus on classification boundaries or regions of high variance.
- Multi-objective Optimization: The quality function can prioritize regions near the estimated Pareto front.
Experimental Results
We evaluated our approach across synthetic benchmarks, real-world active learning tasks, and complex engineering problems.
Synthetic Benchmarks
On standard BO benchmarks (e.g., 4d Shekel, 5d Ackley, 6d Hartmann), the IMP-DPP consistently outperformed traditional baselines. We observed that Sparse GPs using our allocation strategy often achieved better results than even Exact GPs. We hypothesize that by focusing modeling capacity on promising regions, Sparse GPs can effectively "ignore" distracting local minima that might otherwise trap an Exact GP.
Active Learning: Malaria Mapping
In a real-world task involving the mapping of malaria incidence in Nigeria, we utilized a custom quality function for active learning. Our approach achieved 89% accuracy, significantly outperforming the 79% accuracy achieved by the standard CVR approach. This demonstrates the framework's utility in non-optimization sequential decision-making tasks.
High-Dimensional Engineering
We tested the scalability of our method on a 9d Heat Exchanger Design problem, which involves optimizing the weight and effectiveness of an automotive radiator. With only 100 inducing points, our Sparse GP matched the performance of an Exact GP but operated with order-of-magnitude lower computational costs. Crucially, the Sparse GP maintained a constant overhead per step, whereas the Exact GP's costs continued to grow with each new observation.
Discussion and Future Work
Our findings suggest that for sequential decision-making, global modeling accuracy is a secondary concern to task-specific fidelity. The "pixel budget" analogy holds: it is better to have a high-resolution image of the target than a low-resolution image of the entire world.
While the framework is robust, it does require the user to define a sensible quality function $q(z)$. Future research could explore automated ways to learn these functions or extend the method to non-Euclidean spaces, such as discrete molecular structures or graphs. Additionally, applying these allocation strategies to Deep Gaussian Processes presents an exciting frontier for further scaling modeling capacity.
Conclusion
By shifting the focus of inducing point allocation from global uncertainty reduction to task-specific quality, we enable high-throughput Bayesian Optimization that is both fast and precise. This framework provides a scalable, flexible solution for a wide range of complex optimization and active learning problems in science and engineering.
Summary updated