Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes

Cristian Guarnizo, Mauricio A. Álvarez

Introduction

Gaussian Processes (GPs) are powerful tools for non-parametric regression and interpolation. However, standard GPs lack an inherent understanding of the physical laws that often govern real-world data. Latent Force Models (LFMs) bridge this gap by encoding differential equations—such as Newton’s laws or heat transfer equations—directly into the GP's covariance function (the kernel).

While physically grounded, LFMs face a significant computational bottleneck. Constructing their kernels requires solving a convolution integral between a Green’s function (representing the system physics) and a latent force. Analytically, this results in expensive mathematical functions like the Faddeeva function, making LFMs computationally heavy and difficult to scale to large datasets. This paper introduces Random Fourier Response Features (RFRF), a framework that uses frequency-domain approximations to transform these complex integrals into efficient linear operations.

The RFRF Framework

The core innovation of this work is the transition from spatial-domain convolutions to frequency-domain approximations. The authors propose a high-level recipe to break the LFM bottleneck:

  1. Bochner’s Theorem: The latent force's kernel is represented in the frequency domain as an expectation over a probability density.
  2. Decoupling via Laplace Transforms: By moving to the frequency domain, the authors decouple the double integral. They use Laplace transforms to determine how the physical system (the Green's function) responds to a simple complex exponential excitation ($e^{j\lambda t}$).
  3. Monte Carlo Approximation: Instead of solving the full integral, the method takes $S$ random samples from the frequency distribution. This turns the kernel into a simple inner product of "response features."

This approach reduces the computational complexity from $O(D^3N^3)$ to $O(DNQ^2S^2)$, effectively making the scaling linear with respect to the number of observations $N$.

Comparison of exact LFM (ODE1) and RFRF approximations with S=100 and S=10 samples on the Air Temperature dataset.

Experimental Evaluation

The authors evaluated RFRF across three distinct domains:

  • Air Temperature: Time-series data from four sensor locations on the south coast of England.
  • Motion Capture (MOCAP): Joint angle data from "Golf Swing" and "Walking" sequences.
  • SARCOS: A benchmark for multi-output regression involving the inverse dynamics of a seven-degree-of-freedom robot arm.

The "Response" Feature Concept

A key insight is the creation of the "Random Fourier Response Feature" ($v_d$). This feature represents how a specific physical system (defined by an ODE) reacts to a specific frequency. This decouples the dynamics of the system from the statistics of the driving force. Consequently, one can change the latent force's kernel (e.g., from an Squared Exponential to another kernel) simply by changing the sampling distribution $p(\lambda)$, without re-solving the underlying physics.

Key Results

The RFRF approach demonstrated significant improvements in both speed and scalability without sacrificing predictive power:

  • Speedup: On the Golf Swing dataset, the RFRF approach was 3 to 13 times faster than the standard LFM, depending on the number of samples used.
  • Accuracy: With a sufficient number of samples (e.g., $S=100$), the approximation reached or exceeded the accuracy of the exact LFM. In some cases, such as the "Walk" dataset, the RFRF model provided a better fit, likely due to superior numerical stability compared to the complex Faddeeva functions used in exact kernels.
  • Scalability: The linear scaling allowed the model to handle thousands of data points that would typically be prohibitive for standard LFM implementations.
Performance of RFRF on the MOCAP Golf Swing dataset, showing the impact of sample size S on tracking accuracy.

Limitations and Future Work

While RFRF offers a major speed advantage, its quality depends on the number of Monte Carlo samples ($S$). In high-dimensional spaces, a large $S$ might be required, which could diminish speed gains. Additionally, the current derivation assumes stationary latent force kernels.

Future research directions include extending the framework to non-stationary kernels and employing Quasi-Monte Carlo sampling to improve approximation accuracy with fewer samples. The authors also envision integrating these features into Deep GPs or sequential models where physical constraints and high-speed processing are both critical.

Summary updated

BibTeX
@inproceedings{guarnizo2018fast,
  author = {Cristian Guarnizo and Mauricio A. Álvarez},
  title = {{Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=YRDNSvnryp},
}
Figure