Infinite-Dimensional Kalman Filtering Approach to Spatio-Temporal Gaussian Process Regression

Simo Särkkä, Jouni HartikainenAISTATS 2012 · OpenReview

Abstract

We show how spatio-temporal Gaussian process (GP) regression problems (or the equivalent Kriging problems) can be formulated as infinite-dimensional Kalman filtering and Rauch-Tung-Striebel (RTS) smoothing problems, and present a procedure for converting spatio-temporal covariance functions into infinite-dimensional stochastic differential equations (SDEs). The resulting infinite-dimensional SDEs belong to the class of stochastic pseudo-differential equations and can be numerically treated using the methods developed for deterministic counterparts of the equations. The scaling of the computational cost in the proposed approach is linear in the number of time steps as opposed to the cubic scaling of the direct GP regression solution. We also show how separable covariance functions lead to a finite-dimensional Kalman filtering and RTS smoothing problem, present analytical and numerical examples, and discuss numerical methods for computing the solutions.

1. Introduction: The Computational Challenge of Spatio-Temporal GPs

Gaussian Process (GP) regression is a cornerstone of spatial statistics, providing a flexible, non-parametric framework for modeling complex fields. However, when time is introduced as a dimension, standard GP methods encounter a severe "curse of dimensionality." In a traditional GP framework, every observation at every time point is treated as part of a single joint distribution. For a dataset with $T$ time steps, the computational cost of inference scales as $O(T^3)$, rendering long-term climate modeling or high-frequency sensor monitoring computationally infeasible without significant approximations.

While Kalman filtering offers an efficient $O(T)$ alternative for linear time-series modeling, it is traditionally restricted to finite-dimensional state vectors. This paper bridges this gap by formulating spatio-temporal GPs as infinite-dimensional Kalman filtering and Rauch-Tung-Striebel (RTS) smoothing problems.

2. Methodology: From Covariance to SDEs

The core innovation lies in transforming a spatio-temporal GP into an infinite-dimensional state-space model. This is achieved by viewing the spatio-temporal process $f(x, t)$ as the solution to an infinite-dimensional linear Stochastic Differential Equation (SDE).

The authors provide a systematic procedure for this conversion using Fourier transforms and spectral factorization. By analyzing the spectral density of a stationary covariance function, they derive a corresponding (pseudo-)differential operator that describes the evolution of the spatial field over time. This allows the extension of classical Kalman filtering equations to work with these operators. Instead of matrix multiplications, the filter applies differential operators to functions.

Spectral density and corresponding spatio-temporal covariance functions illustrating the conversion process.

To implement this numerically, the infinite-dimensional operators are projected onto a finite basis, such as the eigenfunctions of a Laplacian. This discretization transforms the problem into a high-dimensional but manageable finite Kalman filter, preserving the linear $O(T)$ scaling. This approach allows for handling non-separable kernels—where spatial correlation changes over time—by using pseudo-differential operators.

3. Experimental Evaluation and Results

The framework was evaluated on both simulated data (using the non-separable Cressie & Huang model) and a large-scale real-world dataset of US monthly precipitation and temperature (1895–1997). The real-world analysis focused on a subset of approximately 500 stations around Colorado to compare the Spatio-Temporal GP (STGP) against Independent GPs (IGP) for each month.

The results demonstrate significant gains in both efficiency and accuracy. For a test case with 500 time steps, the Kalman/RTS approach was approximately 100 times faster than the direct GP solution. In terms of predictive performance, the STGP consistently outperformed the IGP models.

Mean Squared Error (MSE) as a function of the number of basis functions (m) for STGP and IGP models across different datasets.

As shown in Figure 3, the STGP models (with different smoothness parameters $\nu$) achieve lower MSE than the IGP as the number of spatial basis functions $m$ increases. Specifically, temporal modeling reduced the MSE by at least 20% for precipitation data and even more for temperature data. The authors also demonstrated that even a 4th-order Taylor expansion of the spectral density is sufficient to capture the essential dynamics of complex non-separable kernels.

4. Discussion and Future Work

By treating time as a dynamic process with inherent causality, the proposed method leverages recursive Bayesian filtering to achieve linear scaling. This "forgetting" of the distant past is the key to its efficiency.

However, challenges remain. Deriving SDEs for arbitrary kernels can be mathematically complex, often involving pseudo-differential operators that require specialized numerical solvers. Additionally, while the method is linear in time, it still scales cubically with the number of spatial basis functions $O(M^3)$. Future research will focus on extending this framework to non-stationary dynamics and further optimizing spatial scaling for extremely high-resolution fields.

Summary updated

Abstract

We show how spatio-temporal Gaussian process (GP) regression problems (or the equivalent Kriging problems) can be formulated as infinite-dimensional Kalman filtering and Rauch-Tung-Striebel (RTS) smoothing problems, and present a procedure for converting spatio-temporal covariance functions into infinite-dimensional stochastic differential equations (SDEs). The resulting infinite-dimensional SDEs belong to the class of stochastic pseudo-differential equations and can be numerically treated using the methods developed for deterministic counterparts of the equations. The scaling of the computational cost in the proposed approach is linear in the number of time steps as opposed to the cubic scaling of the direct GP regression solution. We also show how separable covariance functions lead to a finite-dimensional Kalman filtering and RTS smoothing problem, present analytical and numerical examples, and discuss numerical methods for computing the solutions.
BibTeX
@inproceedings{sarkka2012infinite,
  author = {Simo Särkkä and Jouni Hartikainen},
  title = {{Infinite-Dimensional Kalman Filtering Approach to Spatio-Temporal Gaussian Process Regression}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=ucx1nY0G2E},
}
Figure