Extended and Unscented Gaussian Processes

Daniel M. Steinberg, Edwin V. BonillaNeurIPS 2014 · OpenReview

Introduction

Gaussian Processes (GPs) are a gold standard for non-parametric regression, offering flexible and probabilistically grounded modeling. However, they possess a major Achilles' heel: non-conjugate likelihoods. When the observation $y$ is simply the latent function $f$ plus Gaussian noise, the mathematics are straightforward. But when $y$ is a complex, nonlinear transformation of $f$ (e.g., $y = \sin(f)$ or $y = \text{logistic}(f)$), the posterior distribution becomes mathematically intractable.

This scenario is common in "inversion problems," such as determining the joint torques of a robot arm based on its end-effector position. Traditionally, researchers turn to expensive Markov Chain Monte Carlo (MCMC) methods or complex variational approximations requiring difficult multidimensional integrals. To address this, recent work introduces two new methods—the Extended Gaussian Process (EGP) and the Unscented Gaussian Process (UGP). By applying linearization techniques borrowed from classical filtering (like the Extended and Unscented Kalman Filters) within a variational framework, these methods enable efficient inference even when the relationship between latent variables and observations is a "black-box" or non-differentiable.

Variational Inference via Linearization

The authors frame the solution using Variational Inference, approximating the true, intractable posterior with a simpler Gaussian distribution. To make the mathematics tractable, they linearize the nonlinear likelihood function $g(f)$ around the current estimate of the mean. This yields two distinct algorithms:

  1. Extended GP (EGP): This method uses a first-order Taylor series expansion, requiring the derivative (Jacobian) of the likelihood function. It serves as the GP equivalent of the Iterated Extended Kalman Filter (iEKF).
  2. Unscented GP (UGP): This method employs statistical linearization via the unscented transform. Instead of calculating derivatives, it selects a small set of representative "sigma points," passes them through the nonlinear function, and calculates the statistics of the output. This is the GP equivalent of the Iterated Sigma-Point Kalman Filter (iSPKF).

A key theoretical contribution of this work is proving that the parameter updates in this variational framework are mathematically equivalent to the update equations used in recursive Kalman filtering. This effectively bridges the gap between state-space modeling and Gaussian Processes. Furthermore, by assuming a factorizing likelihood, the authors reduce the number of parameters to be inferred from $O(N^2)$ to $O(N)$, significantly improving computational scalability.

Experimental Validation

The methods were evaluated across two main domains: synthetic inversion problems and real-world binary classification.

Synthetic Inversion Problems

The authors generated latent data and transformed it using various differentiable functions, such as $f^3 + f^2 + f$, $\exp(f)$, $\sin(f)$, and $\tanh(2f)$. To specifically showcase the UGP's derivative-free capabilities, they also tested a non-differentiable function: $g(f) = 2 \times \text{sign}(f) + f^3$.

UGP performance on a non-differentiable synthetic inversion problem.

In these tests, both EGP and UGP consistently outperformed existing variational baselines in terms of Negative Log Predictive Density (NLPD) and Standardized Mean Squared Error (SMSE). The UGP proved remarkably robust, performing nearly as well as the EGP on differentiable functions and succeeding entirely where the EGP would fail due to a lack of gradients.

Real-World Classification

To test practical applicability, the algorithms were applied to a binary handwritten digit classification task using the USPS dataset (specifically distinguishing between the digits '3' and '5').

Classification error rates on the USPS dataset comparing UGP to baselines.

The UGP demonstrated exceptional classification power, achieving the lowest error rate (1.94%). It outperformed specialized GP classifiers (including Laplace, Expectation Propagation, and Variational Bayes approximations) as well as a Support Vector Machine (SVM).

Limitations and Future Directions

While powerful, these methods rely on linearization and are therefore approximations. If the likelihood is extremely "wiggly" or discontinuous within the range of the posterior, the approximation may degrade. Additionally, the authors note that for the UGP, the objective function can occasionally diverge during hyperparameter learning, necessitating careful monitoring or line-search strategies.

Future work will likely focus on extending these techniques to multi-task GPs. This would allow for the handling of inversion problems with multiple inputs and outputs, opening the door to more complex applications like advanced robot kinematics.

Summary updated

Abstract

We present two new methods for inference in Gaussian process (GP) models with general nonlinear likelihoods. Inference is based on a variational framework where a Gaussian posterior is assumed and the likelihood is linearized about the variational posterior mean using either a Taylor series expansion or statistical linearization. We show that the parameter updates obtained by these algorithms are equivalent to the state update equations in the iterative extended and unscented Kalman filters respectively, hence we refer to our algorithms as extended and unscented GPs. The unscented GP treats the likelihood as a 'black-box' by not requiring its derivative for inference, so it also applies to non-differentiable likelihood models. We evaluate the performance of our algorithms on a number of synthetic inversion problems and a binary classification dataset.
BibTeX
@inproceedings{steinberg2014extended,
  author = {Daniel M. Steinberg and Edwin V. Bonilla},
  title = {{Extended and Unscented Gaussian Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=SJNJVd-uZH},
}
Figure