Efficient state-space inference of periodic latent force models

Steven Reece, Siddhartha Ghosh, Alex Rogers, Stephen J. Roberts, Nicholas R. Jennings

TL;DR

This paper introduces a computationally efficient state-space framework for Latent Force Models (LFMs) that can handle periodic and quasi-periodic hidden forces. By using an optimal eigenfunction basis to represent these forces, the authors achieve a massive reduction in error—reducing RMSE to as little as 17% of non-periodic models—while maintaining the $O(T)$ scaling necessary for long time-series data.

What Problem Does This Paper Address?

In many engineering and scientific problems, we know the "physics" of a system (expressed as differential equations), but we don't know all the forces acting on it. Latent Force Models (LFMs) bridge this gap by plugging Gaussian Process (GP) priors into those differential equations.

However, LFMs face a "double bind":

  1. Computational Cost: Standard GP inference scales cubically with the number of time points ($O(T^3)$), making it unusable for long datasets.
  2. The Periodicity Gap: While state-space methods (like Kalman Filters) can speed up inference to $O(T)$, they traditionally only work for simple kernels like Matérn or Squared-Exponential. They struggle with periodic forces—like the daily cycle of the sun or human behavior—which are ubiquitous in real-world data.

Existing workarounds, like the "resonator model," often fail to capture the full prior information of the GP or become computationally prohibitive when trying to learn model parameters.

Methods

The authors propose a novel Linear Basis Model (LBM) approach to represent periodic latent forces within a state-space framework.

  • Eigenfunction Basis: Instead of using arbitrary sines and cosines, they use Kernel Principal Component Analysis (KPCA) and the Nyström approximation to derive the optimal basis functions (eigenfunctions) for any given periodic kernel.
  • State-Space Augmentation: These basis functions are integrated into a Kalman Filter by augmenting the state vector. This allows the model to "track" the weights of these periodic forces in real-time.
  • Handling "Quasi-Periodicity": Real-world cycles aren't perfect; they drift. The authors introduce three "Quasi" models—Continuous (CQM), Step (SQM), and Wiener-step (WQM)—which allow the weights of the periodic basis functions to evolve over time, capturing how a daily routine might shift slightly from day to day.
  • Efficiency: Because it uses a sparse representation (a small number of significant eigenfunctions), the model remains computationally lean.

Datasets

The authors validated their approach on two distinct, real-world applications:

  1. Call Center Arrivals: Real customer arrival rate data from a loan company (collected every 5 minutes over three months). This is a non-linear tracking problem where arrival rates are "quasi-periodic" (similar every Thursday, but not identical).
  2. Home Thermal Dynamics: Internal and external temperature data, along with heater activity, from two real domestic buildings in the UK. This tests the model's ability to predict "day-ahead" temperatures by accounting for periodic residual heat (e.g., solar gain and human activity).
  3. Simulated Thermal Data: Used to rigorously compare the different quasi-periodic kernels against the "resonator" baseline in a controlled environment.

Key Insights and Ideas

  • The "Aha" Moment on Resonators: The authors prove that the existing "resonator model" is actually a special, often sub-optimal case of their eigenfunction approach. By using eigenfunctions, they can encode any periodic GP prior into a resonator-like structure.
  • Optimal vs. Random: Unlike Sparse Spectrum GP (SSGPR) methods that sample frequencies randomly, the eigenfunction approach identifies the most informative frequencies deterministically. This leads to much more accurate posterior variance estimates.
  • Physics + Residuals: You don't need a perfect physical model of a house to predict its temperature. By modeling the "unknown" parts (solar gain) as a periodic latent force, the LFM compensates for the simplicity of the underlying thermal ODE.

Key Results

  • Massive Accuracy Gains: In call center queue tracking, the proposed model reduced the Root Mean Squared Error (RMSE) to 17% of that of non-periodic models.
  • Outperforming Rivals: In thermal modeling, the eigenfunction approach achieved an RMSE that was 45% of the nearest rival (the resonator model).
  • Computational Speed: The training phase for the eigenfunction model is linear in the number of basis functions, whereas the resonator model's training is exponentially complex, making the new approach significantly faster for complex systems.

Limitations and Future Work

  • Static Basis: The current model uses a static basis derived from the prior covariance. The authors suggest that an "adaptable" basis that updates as new evidence arrives (the posterior) would be even more powerful.
  • Control Applications: The Kalman formalism used here is a natural fit for Model Predictive Control (MPC). Future work could involve using these LFMs to optimize home heating systems in real-time to save energy.
  • Non-Gaussian Extensions: While the thermal model uses a Particle Filter to handle binary heater states, the core LFM framework is primarily Gaussian. Extending this to more complex, non-Gaussian likelihoods remains an open area.

Summary updated

Abstract

Latent force models (LFM) are principled approaches to incorporating solutions to differential equations within non-parametric inference methods. Unfortunately, the development and application of LFMs can be inhibited by their computational cost, especially when closed-form solutions for the LFM are unavailable, as is the case in many real world problems where these latent forces exhibit periodic behaviour. Given this, we develop a new sparse representation of LFMs which considerably improves their computational efficiency, as well as broadening their applicability, in a principled way, to domains with periodic or near periodic latent forces. Our approach uses a linear basis model to approximate one generative model for each periodic force. We assume that the latent forces are generated from Gaussian process priors and develop a linear basis model which fully expresses these priors. We apply our approach to model the thermal dynamics of domestic buildings and show that it is effective at predicting day-ahead temperatures within the homes. We also apply our approach within queueing theory in which quasi-periodic arrival rates are modelled as latent forces. In both cases, we demonstrate that our approach can be implemented efficiently using state-space methods which encode the linear dynamic systems via LFMs. Further, we show that state estimates obtained using periodic latent force models can reduce the root mean squared error to 17% of that from non-periodic models and 27% of the nearest rival approach which is the resonator model.
BibTeX
@article{reece2014efficient,
  author = {Steven Reece and Siddhartha Ghosh and Alex Rogers and Stephen J. Roberts and Nicholas R. Jennings},
  title = {{Efficient state-space inference of periodic latent force models}},
  journal = {J. Mach. Learn. Res.},
  year = {2014},
  url = {https://openreview.net/forum?id=Blaf2u3G2C},
}

Loading related content...

Figure