Abstract
Gaussian process (GP) models provide a powerful tool for prediction but are computationally prohibitive using large data sets. In such scenarios, one has to resort to approximate methods. We derive an approximation based on a composite likelihood approach using a general belief updating framework, which leads to a recursive computation of the predictor as well as of learning the hyper-parameters. We then provide an analysis of the derived composite GP model in predictive and information-theoretic terms. Finally, we evaluate the approximation with both synthetic data and a real-world application.
Introduction
Gaussian Processes (GPs) are widely regarded as the "gold standard" for non-parametric regression due to their robust uncertainty quantification. However, their application is severely limited by an $O(N^3)$ computational complexity for training, where $N$ is the number of data points. This cubic scaling stems from the necessity of inverting the covariance matrix, rendering standard GPs impractical for datasets exceeding a few thousand points. While approximations like Sparse GPs (SGP) and Product of Experts exist, they often lack a unified theoretical foundation for quantifying information loss. This paper introduces Composite Gaussian Processes (CGP), a recursive framework designed to enable scalable learning and prediction while providing rigorous performance analysis.
Methodology: The Composite Framework
The CGP framework employs a "divide and conquer" strategy, partitioning the dataset into $K$ manageable segments of size $N_k$. The core of the approach is a composite likelihood function defined as the sum of log-likelihoods of these individual segments, effectively treating them as local experts.
A key innovation is the recursive update mechanism. Using a Bayesian belief-updating framework, the model updates its posterior distribution as each new data block arrives, analogous to a Kalman filter applied to latent functions. For hyper-parameter optimization (e.g., length-scales), the authors propose a weighted average of Maximum Likelihood Estimates (MLEs) from each block. These estimates are weighted by the Fisher Information Matrix (FIM), ensuring that blocks with higher information content exert a greater influence on the final model parameters.
Experimental Evaluation
The CGP model was validated across three distinct scenarios:
- Synthetic Time Series: 4,224 points with periodic patterns to test recursive prediction logic.
- Synthetic Spatial Data: A 2D Gaussian Random Field (GRF) on a $64 \times 64$ grid to compare interpolation quality against standard and Sparse GPs.
- Real-world NOx Prediction: Over 10 years of hourly nitrogen oxide (NOx) measurements, requiring log-transformation to handle highly skewed data.
Performance and Information-Theoretic Analysis
The analysis reveals that CGP segments are effectively "connected" through test points, assuming conditional independence of training blocks given the test data. Information-theoretic analysis using mutual information shows that CGP's performance relative to a full GP depends on whether data segments are redundant or synergistic. If segments contain overlapping information, they are redundant; if they provide more info when combined than the sum of their parts, they are synergistic.
In spatial interpolation tasks, CGP demonstrated remarkable visual fidelity, producing results nearly indistinguishable from the full GP. In contrast, SGP exhibited higher posterior variance and visible artifacts in its covariance structure. For real-world NOx data, CGP achieved a superior balance between accuracy and speed, outperforming SGP in both runtime and Root Mean Square Error (RMSE). By breaking the problem into $K$ blocks, the complexity drops to $O(K \cdot N_k^3)$, allowing the processing of over 5,000 points in less than a second on standard hardware.
Conclusion
CGP offers a scalable and theoretically grounded alternative to standard GPs for large-scale datasets. While the current work assumes pre-defined segmentation, future research will explore optimal data partitioning strategies and extend theoretical bounds for an arbitrary number of segments. The recursive nature of the algorithm makes it particularly suitable for streaming data and environmental monitoring applications.
Summary updated