Introduction
In engineering and signal processing, tracking a "hidden" state from noisy sensor data is a fundamental challenge. While linear systems are effectively handled by the Kalman Filter, real-world systems are frequently nonlinear with unknown underlying mathematical models. Traditional approaches like the Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF) require pre-defined equations, which are often unavailable. This paper introduces GPIL (Gaussian Process Inference and Learning), a non-parametric framework that simultaneously infers hidden states and learns system dynamics and measurement functions directly from noisy observations.
The GPIL Framework
The core of the GPIL algorithm is the integration of Gaussian Processes (GPs) within an Expectation-Maximization (EM) framework. Unlike parametric models that assume a fixed functional form, GPIL models the transition function $f$ and measurement function $g$ as GPs. This allows the system to adapt its complexity to the data while providing a principled way to quantify model uncertainty.
Model Structure and Pseudo-Points
To maintain computational tractability, the GPs are represented using "pseudo training sets." These consist of optimized input-output pairs—$(\alpha, \beta)$ for dynamics and $(\xi, \upsilon)$ for measurements—that serve as anchors for the functions. This parameterization transforms a non-parametric problem into a continuous optimization task suitable for gradient-based methods.
Inference and Learning (E-Step and M-Step)
The algorithm operates through two primary steps:
- The E-Step (Inference): A novel forward-backward smoothing algorithm is employed to estimate the sequence of hidden states. This process accounts for nonlinearities and the inherent uncertainty of the GP models.
- The M-Step (Learning): The GP hyperparameters and pseudo training sets are updated to maximize the likelihood of the observed data. Crucially, GPIL demonstrates that ground-truth state data is not required; the model "discovers" a latent coordinate system that best explains the observations.
Experimental Evaluation
The authors evaluated GPIL against two distinct datasets to test its robustness and adaptability:
- Synthetic Sinusoidal Dynamics: A 1D system defined by $x_t = 3 \sin(3x_{t-1}) + \epsilon$. This high-curvature environment is specifically designed to challenge filters like the EKF, which often fail at sharp turns in state space.
- Whistler Snowfall Data: Real-world historical daily snowfall data from Whistler, BC. This dataset features a "hinge" behavior—snowfall is zero above freezing and positive otherwise—representing a complex, non-smooth measurement function.
Key Results
GPIL consistently outperformed standard filtering techniques and other GP-based methods, particularly in terms of Negative Log Predictive Likelihood (NLL).
Performance on Synthetic Data
In the sinusoidal task, GPIL achieved an NLL of 0.917, significantly outperforming the EKF (1.23) and UKF (4.55). While the EKF and UKF became "overconfident" and produced large errors at high-curvature points, GPIL successfully recovered the ground truth dynamics.
Performance on Real-World Data
On the Whistler snowfall dataset, GPIL reached an NLL of 0.684, vastly superior to the Kalman filter (1.29) and specialized neural network approaches. The model effectively learned the "hinge" nature of the snowfall data without prior knowledge of the underlying physical constraints.
Conclusion and Future Work
GPIL establishes a powerful framework for learning unknown nonlinear dynamics by treating uncertainty as a first-class citizen. By accounting for system noise, measurement noise, and model uncertainty, it provides more reliable predictions than traditional filters.
Future work remains in addressing the computational complexity of GPs for high-dimensional state spaces. Additionally, while the current M-step uses approximations for tractability, incorporating sampling methods like Monte Carlo could potentially further enhance performance at the cost of increased computation time.
Summary updated