Abstract
Latent force models (LFM) are principled approaches to incorporating solutions to differential equations within non-parametric inference methods. However, their computational cost can be prohibitive, especially when closed-form solutions are unavailable—a common issue in real-world problems where latent forces exhibit periodic behavior. This paper introduces a new sparse representation of LFMs that significantly improves computational efficiency and broadens applicability to periodic or near-periodic forces. By using a linear basis model to approximate generative models for periodic forces, the authors develop a framework that fully expresses Gaussian process priors. Applied to domestic building thermal dynamics and queueing theory, the approach demonstrates efficient implementation via state-space methods. Results show that periodic latent force models can reduce root mean squared error (RMSE) to 17% of that from non-periodic models and 27% of the nearest rival approach, the resonator model.
Introduction: The Challenge of Periodicity in LFMs
In many scientific and engineering domains, physical systems are governed by known differential equations but are influenced by unobserved or "latent" forces. Latent Force Models (LFMs) bridge this gap by integrating Gaussian Process (GP) priors into these equations.
Despite their utility, LFMs face a "double bind":
- Computational Cost: Standard GP inference scales cubically ($O(T^3)$) with the number of time points, making it unusable for long datasets.
- The Periodicity Gap: While state-space methods (like Kalman Filters) can speed up inference to $O(T)$, they traditionally struggle with periodic forces—such as solar cycles or human behavior—which are ubiquitous in real-world data.
Existing workarounds, like the "resonator model," often fail to capture the full prior information of the GP or become computationally prohibitive when learning model parameters.
Methodology: Linear Basis Models
The authors propose a novel Linear Basis Model (LBM) approach to represent periodic latent forces within a state-space framework.
Optimal Eigenfunction Basis
Instead of using arbitrary sines and cosines, the framework uses Kernel Principal Component Analysis (KPCA) and the Nyström approximation to derive the optimal basis functions (eigenfunctions) for any given periodic kernel. Unlike Sparse Spectrum GP (SSGPR) methods that sample frequencies randomly, this approach identifies the most informative frequencies deterministically, leading to more accurate posterior variance estimates.
State-Space Augmentation and Quasi-Periodicity
These basis functions are integrated into a Kalman Filter by augmenting the state vector, allowing the model to track the weights of periodic forces in real-time. To handle real-world cycles that "drift," the authors introduce three "Quasi" models:
- Continuous (CQM): Captures smooth transitions in periodic weights.
- Step (SQM): Models abrupt changes in periodic influence.
- Wiener-step (WQM): Combines step changes with a stochastic drift.
These allow the weights of the periodic basis functions to evolve over time, capturing how a daily routine might shift slightly from day to day.
Experimental Validation
The approach was tested on two distinct real-world applications:
- Call Center Arrivals: Modeling customer arrival rates collected every 5 minutes over three months. This non-linear tracking problem involves "quasi-periodic" rates. The proposed model reduced the RMSE to 17% of that of non-periodic models.
- Home Thermal Dynamics: Internal and external temperature data from UK domestic buildings were used to predict "day-ahead" temperatures. By modeling unknown factors like solar gain as periodic latent forces, the LFM compensates for the simplicity of the underlying thermal ODE. The eigenfunction approach achieved an RMSE that was 45% of the nearest rival (the resonator model).
Key Insights and Future Directions
A significant finding is that the existing "resonator model" is a special, often sub-optimal case of this eigenfunction approach. By using eigenfunctions, any periodic GP prior can be encoded into a resonator-like structure. Furthermore, the training phase for the eigenfunction model is linear in the number of basis functions, whereas the resonator model's training is exponentially complex.
Future work includes applying these LFMs to Model Predictive Control (MPC) to optimize home heating systems in real-time and extending the framework to handle more complex, non-Gaussian likelihoods.
Summary updated