Matérn Gaussian processes on Riemannian manifolds

Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, Marc Peter Deisenroth

TL;DR

This paper introduces a constructive framework for building Matérn and squared exponential Gaussian processes (GPs) on compact Riemannian manifolds, such as spheres, tori, and complex 3D meshes. By leveraging the spectral theory of the Laplace–Beltrami operator, the authors transform abstract stochastic partial differential equations into practical kernels that can be trained using standard, scalable machine learning tools like inducing point methods and mini-batching.

What Problem Does This Paper Address?

Gaussian processes are the gold standard for uncertainty quantification in machine learning, but they are traditionally designed for Euclidean spaces ($\mathbb{R}^d$). However, many real-world problems involve data that lives on non-Euclidean surfaces: climate data on the globe (a sphere), robotic joint angles (tori), or textures on 3D models (complex meshes).

A common "naive" fix is to take a standard kernel and replace Euclidean distance with geodesic distance (the shortest path along the surface). Unfortunately, this often results in mathematically invalid kernels that are not positive semi-definite. While researchers previously proposed defining these GPs as solutions to Stochastic Partial Differential Equations (SPDEs), those methods were "non-constructive"—they required solving complex equations numerically for every new data point, making them nearly impossible to scale or use with modern deep learning frameworks.

Methods

The authors bridge the gap between theoretical SPDEs and practical implementation using spectral theory.

  1. The Spectral Kernel: They show that the kernel of a Matérn GP on a manifold can be expressed as an infinite sum involving the eigenvalues ($\lambda_n$) and eigenfunctions ($f_n$) of the Laplace–Beltrami operator (the manifold equivalent of the Laplacian).

    Figure 1: The Mat´ern kernel k1/2(x, ·), defined on a circle, sphere and dragon. The point x is marked with a red dot. The height of the solid line and color, respectively, give the value of the kernel.
  2. Constructive Computation: For simple shapes like spheres or circles, these "eigenpairs" are known analytically (e.g., spherical harmonics). For complex shapes like the "dragon" mesh, they can be computed once upfront using a standard finite element method (FEM) solver.

  3. Scalability: By truncating this infinite sum to $N$ terms, the kernel becomes a finite basis expansion. This allows the use of Sparse Variational Gaussian Processes (SVGP) and Random Fourier Features, enabling the model to handle large datasets via mini-batching and work with non-Gaussian data (like classification) through variational inference.

Datasets

The paper evaluates the method on several geometric settings rather than standard tabular datasets:

  • Analytical Manifolds: The circle ($S^1$), the sphere ($S^d$), and the $d$-dimensional torus ($T^d$).
  • The Dragon Manifold: A complex 3D mesh consisting of over 200,000 triangles, used to demonstrate how the GP respects intricate geometry.
  • Pendulum Dynamics: Synthetic data representing an ideal pendulum, where the state space is a cylinder ($S^1 \times \mathbb{R}$).

Key Insights and Ideas

  • Topology Matters: The naive geodesic approach fails because it ignores the global structure of the space. The spectral approach naturally accounts for "loops"—paths that wrap around the manifold—which is essential for maintaining mathematical validity.

    Figure 2: Visualization of different paths (loops) considered by the periodic summation on a circle.
  • The "DNA" of the Manifold: The eigenfunctions of the Laplace–Beltrami operator act as a natural coordinate system for the manifold. Once you compute these (the "spectral decomposition"), building a GP is just a matter of weighting these basis functions correctly.

  • Unified Smoothness: The authors show that the widely-used squared exponential kernel is simply the limit of the Matérn kernel as smoothness goes to infinity, even in the manifold setting.

Key Results

  • Geometric Awareness: In the dragon mesh experiment, the GP correctly identifies that two points on opposite sides of the snout are "far apart" in manifold distance, even if they are "close" in 3D Euclidean space. This prevents the model from incorrectly smoothing data across physical gaps.

    Figure 5: Visualization of the GP posterior on the dragon mesh. The uncertainty (standard deviation) clearly follows the surface geometry.
  • Dynamical Systems: The model successfully learned the equations of motion for a pendulum on a cylindrical phase space. By respecting the $S^1$ geometry, the GP produced physically consistent trajectories and well-calibrated uncertainty intervals.

    Figure 4: Learned phase diagram and trajectories for the pendulum system.

Limitations and Future Work

  • Compactness Requirement: The current theory is limited to compact manifolds without boundaries. Extending this to non-compact spaces (like an infinite plane with holes) or manifolds with edges remains an open challenge.
  • Truncation Error: The accuracy depends on how many eigenvalues ($N$) are used. For very "rough" functions (low $\nu$), many terms are needed, which increases computational cost.
  • Numerical Pre-processing: For arbitrary meshes, the user must still perform an initial numerical eigen-decomposition, which can be expensive for extremely high-resolution 3D models.

Summary updated

Abstract

Gaussian processes are an effective model class for learning unknown functions, particularly in settings where accurately representing predictive uncertainty is of key importance. Motivated by applications in the physical sciences, the widely-used Matérn class of Gaussian processes has recently been generalized to model functions whose domains are Riemannian manifolds, by re-expressing said processes as solutions of stochastic partial differential equations. In this work, we propose techniques for computing the kernels of these processes on compact Riemannian manifolds via spectral theory of the Laplace-Beltrami operator in a fully constructive manner, thereby allowing them to be trained via standard scalable techniques such as inducing point methods. We also extend the generalization from the Matérn to the widely-used squared exponential Gaussian process. By allowing Riemannian Matérn Gaussian processes to be trained using well-understood techniques, our work enables their use in mini-batch, online, and non-conjugate settings, and makes them more accessible to machine learning practitioners.
BibTeX
@inproceedings{borovitskiy2020matern,
  author = {Viacheslav Borovitskiy and Alexander Terenin and Peter Mostowsky and Marc Peter Deisenroth},
  title = {{Matérn Gaussian processes on Riemannian manifolds}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=gxzBqc4bqnV},
}
Figure