Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes

Cristian Guarnizo, Mauricio A. Álvarez

Introduction

Latent Force Models (LFMs) are a powerful class of physics-informed Gaussian Processes (GPs) that incorporate physical laws, such as differential equations, to model complex systems. However, their computational complexity scales cubically ($O(D^3N^3)$) with the number of outputs and data points, primarily due to the double integral bottleneck in calculating the covariance matrix. This paper introduces a method to significantly speed up LFMs using Random Fourier Features (RFF), reducing complexity to linear relative to the number of observations.

Methodology: Fast Kernel Approximation

The proposed approach leverages Random Fourier Features to approximate the latent force kernel, transforming the daunting convolution integrals into a more manageable form.

  1. Bochner’s Theorem: Stationary kernels are represented as the Fourier transform of a probability density.
  2. Decoupling Integrals: By replacing the latent force kernel with its RFF approximation, the double integral is split into two simpler, separate integrals.
  3. Random Fourier Response Features (RFRF): These integrals represent the physical system's response to complex exponential excitations and are solved analytically using Laplace transforms.
  4. Monte Carlo Approximation: The final kernel is computed as an inner product of these response features, averaged over $S$ random samples.

This shift reduces computational complexity to $O(DNQ^2S^2)$, enabling linear scaling with respect to the number of observations ($N$) through the use of the matrix inversion lemma.

Experimental Evaluation

The authors evaluated the RFRF approach on three distinct datasets:

  • Air Temperature Data: Time series from four sensor locations on the south coast of England (Bramblemet, Sotonmet, Cambermet, and Chimet).
  • Motion Capture (MOCAP): Joint angle data from human movements, specifically "Golf Swing" and "Walk" sequences.
  • SARCOS: A high-dimensional dataset involving joint torques and positions of a redundant seven-DOF robot arm.

Results and Discussion

The RFRF approach demonstrates a significant improvement in computational efficiency without sacrificing predictive accuracy. By bypassing numerical integration through the use of transfer functions evaluated at specific frequencies, the model becomes much more scalable.

Comparison of exact LFM (ODE1) and RFRF approximations (S100, S10) on air temperature data.

On the Air Temperature task, the RFRF approach with $S=50$ or $100$ samples achieved better predictive performance than the standard LFM while requiring significantly less computation time. As shown in the table below, the approximation quality improves as the number of samples $S$ increases, with $S=50$ and $S=100$ providing competitive NMSE and NLPD values compared to the exact ODE1 model.

Performance metrics and computation time for different kernel approximations on the Air Temperature dataset.

In MOCAP experiments, the RFRF approximation ($S=100$) was over 3x faster than the standard second-order LFM while maintaining comparable accuracy. On the high-dimensional SARCOS dataset, the proposed method was nearly 4x faster than state-of-the-art convolved multiple-output GP models.

Conclusion

This framework provides a scalable and flexible solution for physics-informed machine learning by transforming complex convolution integrals into efficient feature-based approximations. While the quality of the approximation depends on the number of Monte Carlo samples, the linear scaling with respect to observations allows LFMs to be applied to much larger datasets than previously possible. Future work includes exploring non-stationary kernels and more efficient sampling methods like Quasi-Monte Carlo to further optimize the approximation.

Summary updated

Abstract

A latent force model is a Gaussian process with a covariance function inspired by a differential operator. Such covariance function is obtained by performing convolution integrals between Green's functions associated to the differential operators, and covariance functions associated to latent functions. In the classical formulation of latent force models, the covariance functions are obtained analytically by solving a double integral, leading to expressions that involve numerical solutions of different types of error functions. In consequence, the covariance matrix calculation is considerably expensive, because it requires the evaluation of one or more of these error functions. In this paper, we use random Fourier features to approximate the solution of these double integrals obtaining simpler analytical expressions for such covariance functions. We show experimental results using ordinary differential operators and provide an extension to build general kernel functions for convolved multiple output Gaussian processes.
BibTeX
@inproceedings{guarnizo2018fast,
  author = {Cristian Guarnizo and Mauricio A. Álvarez},
  title = {{Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=YRDNSvnryp},
}
Figure