Efficient State-Space Inference of Periodic Latent Force Models

Steven Reece, Siddhartha Ghosh, Alex Rogers, Stephen Roberts, Nicholas R. Jennings

Abstract

Latent force models (LFM) are principled approaches to incorporating solutions to differential equations within non-parametric inference methods. However, their computational cost can be prohibitive, especially when closed-form solutions are unavailable—a common issue in real-world problems where latent forces exhibit periodic behavior. This paper introduces a new sparse representation of LFMs that significantly improves computational efficiency and broadens applicability to periodic or near-periodic forces. By using a linear basis model to approximate generative models for periodic forces, the authors develop a framework that fully expresses Gaussian process priors. Applied to domestic building thermal dynamics and queueing theory, the approach demonstrates efficient implementation via state-space methods. Results show that periodic latent force models can reduce root mean squared error (RMSE) to 17% of that from non-periodic models and 27% of the nearest rival approach, the resonator model.

Introduction: The Challenge of Periodicity in LFMs

In many scientific and engineering domains, physical systems are governed by known differential equations but are influenced by unobserved or "latent" forces. Latent Force Models (LFMs) bridge this gap by integrating Gaussian Process (GP) priors into these equations.

Despite their utility, LFMs face a "double bind":

  1. Computational Cost: Standard GP inference scales cubically ($O(T^3)$) with the number of time points, making it unusable for long datasets.
  2. The Periodicity Gap: While state-space methods (like Kalman Filters) can speed up inference to $O(T)$, they traditionally struggle with periodic forces—such as solar cycles or human behavior—which are ubiquitous in real-world data.

Existing workarounds, like the "resonator model," often fail to capture the full prior information of the GP or become computationally prohibitive when learning model parameters.

Methodology: Linear Basis Models

The authors propose a novel Linear Basis Model (LBM) approach to represent periodic latent forces within a state-space framework.

Optimal Eigenfunction Basis

Instead of using arbitrary sines and cosines, the framework uses Kernel Principal Component Analysis (KPCA) and the Nyström approximation to derive the optimal basis functions (eigenfunctions) for any given periodic kernel. Unlike Sparse Spectrum GP (SSGPR) methods that sample frequencies randomly, this approach identifies the most informative frequencies deterministically, leading to more accurate posterior variance estimates.

State-Space Augmentation and Quasi-Periodicity

These basis functions are integrated into a Kalman Filter by augmenting the state vector, allowing the model to track the weights of periodic forces in real-time. To handle real-world cycles that "drift," the authors introduce three "Quasi" models:

  • Continuous (CQM): Captures smooth transitions in periodic weights.
  • Step (SQM): Models abrupt changes in periodic influence.
  • Wiener-step (WQM): Combines step changes with a stochastic drift.

These allow the weights of the periodic basis functions to evolve over time, capturing how a daily routine might shift slightly from day to day.

Experimental Validation

The approach was tested on two distinct real-world applications:

  1. Call Center Arrivals: Modeling customer arrival rates collected every 5 minutes over three months. This non-linear tracking problem involves "quasi-periodic" rates. The proposed model reduced the RMSE to 17% of that of non-periodic models.
  2. Home Thermal Dynamics: Internal and external temperature data from UK domestic buildings were used to predict "day-ahead" temperatures. By modeling unknown factors like solar gain as periodic latent forces, the LFM compensates for the simplicity of the underlying thermal ODE. The eigenfunction approach achieved an RMSE that was 45% of the nearest rival (the resonator model).

Key Insights and Future Directions

A significant finding is that the existing "resonator model" is a special, often sub-optimal case of this eigenfunction approach. By using eigenfunctions, any periodic GP prior can be encoded into a resonator-like structure. Furthermore, the training phase for the eigenfunction model is linear in the number of basis functions, whereas the resonator model's training is exponentially complex.

Future work includes applying these LFMs to Model Predictive Control (MPC) to optimize home heating systems in real-time and extending the framework to handle more complex, non-Gaussian likelihoods.

Summary updated

Abstract

Latent force models (LFM) are principled approaches to incorporating solutions to differential equations within non-parametric inference methods. Unfortunately, the development and application of LFMs can be inhibited by their computational cost, especially when closed-form solutions for the LFM are unavailable, as is the case in many real world problems where these latent forces exhibit periodic behaviour. Given this, we develop a new sparse representation of LFMs which considerably improves their computational efficiency, as well as broadening their applicability, in a principled way, to domains with periodic or near periodic latent forces. Our approach uses a linear basis model to approximate one generative model for each periodic force. We assume that the latent forces are generated from Gaussian process priors and develop a linear basis model which fully expresses these priors. We apply our approach to model the thermal dynamics of domestic buildings and show that it is effective at predicting day-ahead temperatures within the homes. We also apply our approach within queueing theory in which quasi-periodic arrival rates are modelled as latent forces. In both cases, we demonstrate that our approach can be implemented efficiently using state-space methods which encode the linear dynamic systems via LFMs. Further, we show that state estimates obtained using periodic latent force models can reduce the root mean squared error to 17% of that from non-periodic models and 27% of the nearest rival approach which is the resonator model.
BibTeX
@article{reece2014efficient,
  author = {Steven Reece and Siddhartha Ghosh and Alex Rogers and Stephen Roberts and Nicholas R. Jennings},
  title = {{Efficient State-Space Inference of Periodic Latent Force Models}},
  journal = {J. Mach. Learn. Res.},
  year = {2014},
  url = {https://openreview.net/forum?id=Blaf2u3G2C},
}
Figure