Introduction
Latent Force Models (LFMs) are a powerful class of physics-informed Gaussian Processes (GPs) that incorporate physical laws, such as differential equations, to model complex systems. However, their computational complexity scales cubically ($O(D^3N^3)$) with the number of outputs and data points, primarily due to the double integral bottleneck in calculating the covariance matrix. This paper introduces a method to significantly speed up LFMs using Random Fourier Features (RFF), reducing complexity to linear relative to the number of observations.
Methodology: Fast Kernel Approximation
The proposed approach leverages Random Fourier Features to approximate the latent force kernel, transforming the daunting convolution integrals into a more manageable form.
- Bochner’s Theorem: Stationary kernels are represented as the Fourier transform of a probability density.
- Decoupling Integrals: By replacing the latent force kernel with its RFF approximation, the double integral is split into two simpler, separate integrals.
- Random Fourier Response Features (RFRF): These integrals represent the physical system's response to complex exponential excitations and are solved analytically using Laplace transforms.
- Monte Carlo Approximation: The final kernel is computed as an inner product of these response features, averaged over $S$ random samples.
This shift reduces computational complexity to $O(DNQ^2S^2)$, enabling linear scaling with respect to the number of observations ($N$) through the use of the matrix inversion lemma.
Experimental Evaluation
The authors evaluated the RFRF approach on three distinct datasets:
- Air Temperature Data: Time series from four sensor locations on the south coast of England (Bramblemet, Sotonmet, Cambermet, and Chimet).
- Motion Capture (MOCAP): Joint angle data from human movements, specifically "Golf Swing" and "Walk" sequences.
- SARCOS: A high-dimensional dataset involving joint torques and positions of a redundant seven-DOF robot arm.
Results and Discussion
The RFRF approach demonstrates a significant improvement in computational efficiency without sacrificing predictive accuracy. By bypassing numerical integration through the use of transfer functions evaluated at specific frequencies, the model becomes much more scalable.
On the Air Temperature task, the RFRF approach with $S=50$ or $100$ samples achieved better predictive performance than the standard LFM while requiring significantly less computation time. As shown in the table below, the approximation quality improves as the number of samples $S$ increases, with $S=50$ and $S=100$ providing competitive NMSE and NLPD values compared to the exact ODE1 model.
In MOCAP experiments, the RFRF approximation ($S=100$) was over 3x faster than the standard second-order LFM while maintaining comparable accuracy. On the high-dimensional SARCOS dataset, the proposed method was nearly 4x faster than state-of-the-art convolved multiple-output GP models.
Conclusion
This framework provides a scalable and flexible solution for physics-informed machine learning by transforming complex convolution integrals into efficient feature-based approximations. While the quality of the approximation depends on the number of Monte Carlo samples, the linear scaling with respect to observations allows LFMs to be applied to much larger datasets than previously possible. Future work includes exploring non-stationary kernels and more efficient sampling methods like Quasi-Monte Carlo to further optimize the approximation.
Summary updated