Abstract
Despite the success of Gaussian processes (GPs) in modelling spatial stochastic processes, dealing with large datasets is still challenging. The problem arises by the need to invert a potentially large covariance matrix during inference. In this paper we address the complexity problem by constructing a new stationary covariance function (Mercer kernel) that naturally provides a sparse covariance matrix. The sparseness of the matrix is defined by hyperparameters optimised during learning. The new covariance function enables exact GP inference and performs comparatively to the squared-exponential one, at a lower computational cost. This allows the application of GPs to large-scale problems such as ore grade prediction in mining or 3D surface modelling. Experiments show that using the proposed covariance function, very sparse covariance matrices are normally obtained which can be effectively used for faster inference and less memory usage.
Introduction
Gaussian Processes (GPs) are considered a gold standard for spatial regression and stochastic modeling due to their robust uncertainty quantification. However, they suffer from a well-known scaling bottleneck: inference requires the inversion of a covariance matrix, an operation with $O(N^3)$ time complexity and $O(N^2)$ space complexity for $N$ data points. For modern datasets exceeding tens of thousands of points, standard GP inference becomes computationally intractable.
To circumvent this, researchers typically employ approximations, such as using a small set of "inducing points" to represent the global dataset. While these methods reduce complexity, they often introduce discontinuities or lose fine-grained local details. This paper proposes a different path: Exact Sparse Gaussian Processes (ESGPs). Instead of approximating the inference process, the authors introduce a novel covariance function that is inherently sparse, allowing for exact inference using efficient sparse linear algebra.
Methodology: The Sparse Covariance Function
The core innovation is the development of a stationary, "compactly supported" covariance function. Unlike standard kernels like the Squared Exponential (SqExp), which decay towards zero but never reach it, the proposed kernel becomes exactly zero once the distance between two points exceeds a learned threshold $l$.
To ensure mathematical validity (positive semi-definiteness), the kernel is derived by convolving a squared-cosine basis function with itself. This construction results in a kernel that is four times continuously differentiable ($C^4$), ensuring that the resulting GP realizations are smooth and physically plausible. Crucially, the cutoff distance $l$ is treated as a hyperparameter and optimized alongside noise levels by maximizing the marginal likelihood. This allows the model to automatically discover the optimal "radius of influence" for data points based on the underlying spatial correlation of the dataset.
Experimental Evaluation
The authors evaluated the ESGP framework across three distinct scenarios:
- Artificial 1D Dataset: Used to qualitatively compare the ESGP against state-of-the-art sparse approximation methods.
- Rainfall Dataset (SIC): A standard 2D geostatistics benchmark with 467 measurements.
- Iron Ore Dataset: A large-scale 3D dataset from an open-pit mine in Western Australia, comprising approximately 17,000 samples of iron concentration.
Results and Discussion
The experimental results highlight the efficiency and accuracy of the sparse kernel approach. In the 1D toy problem, the ESGP provided a significantly more realistic and smoother fit than hybrid local/global approximation methods, which often exhibit visible discontinuities where local models transition.
In the large-scale mining application, the method demonstrated its true power. For the 17,000-point dataset, the optimized model produced covariance matrices where up to 90% of the elements were zero. This extreme sparsity enabled exact inference that was not only computationally feasible but also significantly faster than standard GP implementations. For a dataset of 4,000 points, the sparse GP outperformed the standard GP as long as the matrix was at least 30% sparse—a threshold easily met in practice. Furthermore, the predictive accuracy of the ESGP remained nearly identical to that of the Squared Exponential kernel, proving that sparsity can be achieved without sacrificing performance.
Conclusion
This work demonstrates that intrinsic sparsity is a viable and powerful alternative to traditional GP approximations. By designing a kernel that naturally drops to zero, the authors enable exact inference on large datasets using sparse matrix libraries. While the current focus is on 3D spatial modeling, the proposed sparse kernel is a valid Mercer kernel that could potentially benefit other kernel-based methods like Support Vector Machines. Future research will explore the scalability of this approach in higher-dimensional feature spaces where the curse of dimensionality may impact the effectiveness of compact support.
Summary updated