Introduction
Gaussian Processes (GPs) are a gold standard for non-parametric regression, offering flexible and probabilistically grounded modeling. However, they possess a major Achilles' heel: non-conjugate likelihoods. When the observation $y$ is simply the latent function $f$ plus Gaussian noise, the mathematics are straightforward. But when $y$ is a complex, nonlinear transformation of $f$ (e.g., $y = \sin(f)$ or $y = \text{logistic}(f)$), the posterior distribution becomes mathematically intractable.
This scenario is common in "inversion problems," such as determining the joint torques of a robot arm based on its end-effector position. Traditionally, researchers turn to expensive Markov Chain Monte Carlo (MCMC) methods or complex variational approximations requiring difficult multidimensional integrals. To address this, recent work introduces two new methods—the Extended Gaussian Process (EGP) and the Unscented Gaussian Process (UGP). By applying linearization techniques borrowed from classical filtering (like the Extended and Unscented Kalman Filters) within a variational framework, these methods enable efficient inference even when the relationship between latent variables and observations is a "black-box" or non-differentiable.
Variational Inference via Linearization
The authors frame the solution using Variational Inference, approximating the true, intractable posterior with a simpler Gaussian distribution. To make the mathematics tractable, they linearize the nonlinear likelihood function $g(f)$ around the current estimate of the mean. This yields two distinct algorithms:
- Extended GP (EGP): This method uses a first-order Taylor series expansion, requiring the derivative (Jacobian) of the likelihood function. It serves as the GP equivalent of the Iterated Extended Kalman Filter (iEKF).
- Unscented GP (UGP): This method employs statistical linearization via the unscented transform. Instead of calculating derivatives, it selects a small set of representative "sigma points," passes them through the nonlinear function, and calculates the statistics of the output. This is the GP equivalent of the Iterated Sigma-Point Kalman Filter (iSPKF).
A key theoretical contribution of this work is proving that the parameter updates in this variational framework are mathematically equivalent to the update equations used in recursive Kalman filtering. This effectively bridges the gap between state-space modeling and Gaussian Processes. Furthermore, by assuming a factorizing likelihood, the authors reduce the number of parameters to be inferred from $O(N^2)$ to $O(N)$, significantly improving computational scalability.
Experimental Validation
The methods were evaluated across two main domains: synthetic inversion problems and real-world binary classification.
Synthetic Inversion Problems
The authors generated latent data and transformed it using various differentiable functions, such as $f^3 + f^2 + f$, $\exp(f)$, $\sin(f)$, and $\tanh(2f)$. To specifically showcase the UGP's derivative-free capabilities, they also tested a non-differentiable function: $g(f) = 2 \times \text{sign}(f) + f^3$.
In these tests, both EGP and UGP consistently outperformed existing variational baselines in terms of Negative Log Predictive Density (NLPD) and Standardized Mean Squared Error (SMSE). The UGP proved remarkably robust, performing nearly as well as the EGP on differentiable functions and succeeding entirely where the EGP would fail due to a lack of gradients.
Real-World Classification
To test practical applicability, the algorithms were applied to a binary handwritten digit classification task using the USPS dataset (specifically distinguishing between the digits '3' and '5').
The UGP demonstrated exceptional classification power, achieving the lowest error rate (1.94%). It outperformed specialized GP classifiers (including Laplace, Expectation Propagation, and Variational Bayes approximations) as well as a Support Vector Machine (SVM).
Limitations and Future Directions
While powerful, these methods rely on linearization and are therefore approximations. If the likelihood is extremely "wiggly" or discontinuous within the range of the posterior, the approximation may degrade. Additionally, the authors note that for the UGP, the objective function can occasionally diverge during hyperparameter learning, necessitating careful monitoring or line-search strategies.
Future work will likely focus on extending these techniques to multi-task GPs. This would allow for the handling of inversion problems with multiple inputs and outputs, opening the door to more complex applications like advanced robot kinematics.
Summary updated