Gaussian Processes (GPs) are the gold standard for non-parametric surrogate modeling, active learning, and Bayesian optimization. They provide elegant, exact uncertainty quantification, allowing algorithms to intelligently balance exploration and exploitation. However, GPs suffer from a severe vulnerability: they break down rapidly as the dimensionality of the input space grows.
To understand why, we must look at the geometry of high-dimensional spaces and how GPs interpret distance.
The Intuition Behind the Breakdown
Standard Gaussian Process regression relies on a kernel function (such as the Radial Basis Function or Matérn kernel) to measure the similarity between points. These kernels assume that points close to each other in the input space will have highly correlated outputs.
In low dimensions, this intuition works perfectly. But as the number of dimensions increases, the volume of the space grows exponentially, and the geometry of the data changes counterintuitively:
- Distance Collapse: In high-dimensional spaces, the Euclidean distance between any two points tends to converge to the same average value. Every point becomes almost equally far from every other point.
- Sparsity: Data points become isolated in vast empty expanses. A kernel matrix filled with nearly identical, large distances becomes flat or ill-conditioned, losing its ability to differentiate between regions of high and low uncertainty.
Practically, as noted by Acerbi in simulation-based science and Huggins et al. in variational inference, standard GP surrogates hit a hard performance bottleneck around 15 to 20 dimensions. Beyond this threshold, the computational cost of fitting the surrogate scales cubically ($O(N^3)$), while its predictive accuracy degrades precipitously.
To scale GPs to dozens, hundreds, or thousands of dimensions, researchers have developed several core strategies that bypass this geometric trap.
Strategy 1: Additive Structures and Kernel Decomposition
One of the most effective ways to evade the curse of dimensionality is to assume that the high-dimensional target function can be broken down into a sum of lower-dimensional functions. If a 100-dimensional function can be approximated as a sum of univariate or bivariate functions, we no longer need to calculate distances across all 100 dimensions simultaneously.
This is the core mechanism behind additive Gaussian Processes. Kandasamy et al. leverage additive structures to scale GP-based bandit optimization, allowing the algorithm to handle higher-dimensional search spaces by optimizing over lower-dimensional components independently.
Similarly, López-Lopera et al. combine an additive model structure with a sequential dimension reduction algorithm called MaxMod. This approach allows them to enforce physical constraints—such as monotonicity—across input spaces with up to 1,000 dimensions, proving that prior physical knowledge can be preserved without succumbing to exponential complexity.
Strategy 2: Projection Pursuit and Active Subspaces
Instead of assuming the function decomposes along the existing coordinate axes, another approach is to find a low-dimensional linear or non-linear subspace that captures the majority of the function's variation.
The authors of Scaling Multidimensional Gaussian Processes using Projected Additive Approximations introduce Projected Additive Gaussian Process Regression (PPGPR). This method pairs projection pursuit with state-space GP techniques to achieve linear $O(N)$ computational scaling, bypassing the cubic bottleneck while maintaining competitive accuracy in high dimensions.
In a similar vein, Cai et al. propose a Bayesian inverse regression framework for supervised dimension reduction. By identifying a low-dimensional "slice" of the data that retains the predictive power of thousands of input predictors, they make GP modeling feasible for small datasets with high-dimensional features.
Strategy 3: Shifting Focus to the Output Space
In tasks like Multi-Objective Bayesian Optimization (MOO), the input space might be incredibly complex and high-dimensional, but the objective space or the target Pareto front lives in a much lower-dimensional output space.
Deshwal et al. exploit this asymmetry through the Output Space Entropy Search framework. Instead of trying to map the entire high-dimensional input space perfectly—which becomes a major bottleneck as dimensions reach the hundreds—their acquisition functions maximize information gain directly about the optimal Pareto front in the output space. This drastically reduces the predictive burden on the underlying surrogate model.
Strategy 4: Intelligent Pre-Calibration
When dealing with complex physical simulations or climate models where parameters can number in the hundreds or thousands, training a GP directly on the raw parameter space is impossible. A powerful workaround is to use a multi-stage framework that filters or calibrates the parameter space before the GP is ever deployed.
Cleary et al. introduce the "Calibrate, Emulate, Sample" (CES) framework to solve this. They use Ensemble Kalman methods during an initial calibration phase to intelligently locate and restrict the parameter space to high-probability regions. The GP surrogate is then trained only on this reduced, high-yield subspace, bypassing the performance degradation that usually occurs when trying to emulate thousands of parameters blindly.
Key Foundational Work
On the Robustness of Temporal Properties for Stochastic Models (2013) — contributed 2 concept elements to this direction.