Sparse Gaussian Processes with Spherical Harmonic Features

Vincent Dutordoir, Nicolas Durrande, James Hensman

Abstract

We introduce a new class of inter-domain variational Gaussian processes (GP) where data is mapped onto the unit hypersphere in order to use spherical harmonic representations. Our inference scheme is comparable to variational Fourier features, but it does not suffer from the curse of dimensionality, and leads to diagonal covariance matrices between inducing variables. This enables a speed-up in inference, because it bypasses the need to invert large covariance matrices. Our experiments show that our model is able to fit a regression model for a dataset with 6 million entries two orders of magnitude faster compared to standard sparse GPs, while retaining state of the art accuracy. We also demonstrate competitive performance on classification with non-conjugate likelihoods.

Introduction

Gaussian Processes (GPs) are powerful non-parametric models known for their high-quality uncertainty estimates. However, their computational cost scales cubically ($O(N^3)$) with the number of data points $N$, making them impractical for large datasets. Sparse GPs address this by approximating the full process using $M$ "inducing points." While effective, standard sparse GPs often rely on local kernels that struggle in high-dimensional spaces, requiring a vast number of points to cover the input domain.

Previous attempts to use global basis functions, such as Variational Fourier Features (VFF), offer a solution but face their own scaling issues in higher dimensions due to the complexity of multi-dimensional feature construction. This paper introduces VISH (Variational Inference with Spherical Harmonics), which leverages the global properties of spherical harmonics to achieve efficient, high-dimensional GP inference.

The VISH Framework

The core innovation of VISH lies in its three-step pipeline designed to simplify the computational burden of GP inference.

1. Hypersphere Mapping

The first step involves mapping the input data onto a unit hypersphere ($\mathbb{S}^{d-1}$). This is achieved by appending a "bias" constant to the input features and then projecting the resulting vector. This geometric transformation is not merely a computational trick; it is inspired by the arc-cosine kernel, which describes the behavior of infinite-width ReLU neural networks.

Mapping input data onto the unit hypersphere by appending a bias term.

2. Zonal Kernels and Spherical Harmonics

Once mapped to the sphere, the GP is defined using "zonal kernels"—functions that depend only on the angular distance between points. The authors then project the GP onto spherical harmonics rather than using physical inducing points.

Spherical harmonics are the eigenfunctions of the Laplace-Beltrami operator on the sphere. This mathematical alignment ensures that the covariance matrix between inducing variables ($K_{uu}$) is exactly diagonal. In GP inference, a diagonal $K_{uu}$ eliminates the need for expensive matrix inversions, reducing the complexity of a major bottleneck from $O(M^3)$ to $O(M)$.

Empirical Evaluation

The authors evaluated VISH against standard Stochastic Variational GPs (SVGP) and VFF across several benchmarks.

Toy Data: The Banana Dataset

To visualize how VISH captures complex decision boundaries, the authors used the 2D "Banana" classification dataset. As the number of inducing variables $M$ increases, the model's fit improves significantly, and the Evidence Lower Bound (ELBO) converges rapidly.

VISH performance on the 2D Banana classification dataset for varying numbers of inducing variables M, showing decision boundaries and ELBO convergence.

Large-Scale Performance

On the Airline Delay dataset (6 million records, 8 dimensions), VISH completed inference in just 41 seconds, compared to approximately 15 minutes for SVGP—a speedup of over 20x. In other configurations, the authors report speedups of up to two orders of magnitude. Crucially, this efficiency does not come at the cost of accuracy; VISH remained competitive with state-of-the-art methods in terms of Mean Squared Error (MSE) and Negative Log Predictive Density (NLPD).

Conclusion

VISH represents a significant step forward in making Gaussian Processes scalable to massive, high-dimensional datasets. By utilizing the global influence of spherical harmonics and the computational simplicity of diagonal covariance matrices, it bypasses the traditional bottlenecks of sparse GP inference. While currently limited by numerical stability in dimensions higher than 9, VISH demonstrates that global basis functions can effectively push the boundaries of GP applications.

Summary updated

Abstract

We introduce a new class of inter-domain variational Gaussian processes (GP) where data is mapped onto the unit hypersphere in order to use spherical harmonic representations. Our inference scheme is comparable to variational Fourier features, but it does not suffer from the curse of dimensionality, and leads to diagonal covariance matrices between inducing variables. This enables a speed-up in inference, because it bypasses the need to invert large covariance matrices. Our experiments show that our model is able to fit a regression model for a dataset with 6 million entries two orders of magnitude faster compared to standard sparse GPs, while retaining state of the art accuracy. We also demonstrate competitive performance on classification with non-conjugate likelihoods.
BibTeX
@inproceedings{dutordoir2020sparse,
  author = {Vincent Dutordoir and Nicolas Durrande and James Hensman},
  title = {{Sparse Gaussian Processes with Spherical Harmonic Features}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=XGxSBARv8H},
}
Figure