Unscented Gaussian Process Latent Variable Model: learning from uncertain inputs with intractable kernels

Daniel Augusto R. M. A. de Souza, César Lincoln C. Mattos, João Paulo Pordeus Gomes

Introduction

Gaussian Process Latent Variable Models (GPLVMs) are a cornerstone of non-linear dimensionality reduction and uncertainty modeling. By treating inputs as latent variables, they allow for flexible mapping from low-dimensional spaces to high-dimensional observations. However, the Bayesian formulation of GPLVMs—which accounts for uncertainty in the latent space—is traditionally constrained by the requirement for analytical tractability. Specifically, the marginal likelihood involves integrals (known as $\Psi$-statistics) that are only solvable for a limited subset of kernels, such as the Radial Basis Function (RBF). This "RBF monopoly" restricts the model's ability to capture complex, non-stationary, or periodic patterns that require more expressive kernels like Matérn or Multilayer Perceptron (MLP) kernels.

The Computational Bottleneck

When analytical solutions are unavailable, researchers typically resort to numerical approximations:

  1. Gauss-Hermite (GH) Quadrature: While deterministic and accurate, GH suffers from the "curse of dimensionality." Its computational cost grows exponentially ($H^D$) with the latent dimension $D$, rendering it impractical for $D > 10$.
  2. Monte Carlo (MC) Sampling: MC methods scale linearly with dimensionality but introduce stochastic noise into the objective function. This noise necessitates the use of stochastic gradient descent (SGD), which is often slower to converge and more sensitive to hyperparameters than deterministic second-order optimizers like L-BFGS-B.

The Unscented Transformation Approach

This paper proposes the Unscented Transformation (UT) as a deterministic alternative for approximating the intractable integrals in Bayesian GPLVMs. Originally developed for the Unscented Kalman Filter, UT approximates the distribution of a non-linear transformation by propagating a small, fixed set of "sigma points."

For a $D$-dimensional latent space, UT requires only $2D+1$ sigma points. These points are chosen deterministically to capture the mean and covariance of the latent distribution. This approach offers three primary advantages:

  • Linear Scalability: The number of function evaluations grows linearly with the latent dimension, bypassing the exponential bottleneck of GH quadrature.
  • Deterministic Objective: Unlike Monte Carlo methods, UT provides a smooth, deterministic objective function. This enables the use of efficient quasi-Newton optimization methods (e.g., BFGS), which typically achieve faster convergence and higher precision.
  • Kernel Flexibility: UT allows for the seamless integration of any kernel function, enabling the use of Matérn, MLP, or Spectral Mixture kernels in a Bayesian framework.
Figure 1: Computational efficiency of UT compared to GH and MC methods across varying input dimensions.

Experimental Evaluation

The authors evaluated the UT-GPLVM across several benchmarks to demonstrate its efficiency and expressive power.

Dimensionality Reduction: Oil Flow

On the classic Oil Flow dataset (12 attributes, 1,000 observations), the UT-GPLVM with a Matérn 3/2 kernel achieved 100% 1-Nearest Neighbor (1-NN) accuracy in the latent space. This outperformed both the analytical RBF version (98%) and the GH-based Matérn version (95%). Notably, UT achieved these results with significantly fewer function evaluations than GH as the latent dimensionality increased.

High-Dimensional Structure: USPS Digits

The USPS dataset, consisting of 7,000 handwritten digit images, highlights the benefit of expressive kernels. Using an MLP kernel—made practical by the UT approach—the model improved classification accuracy from ~36% (with RBF) to ~69%. As shown in the latent space visualizations, the MLP kernel provides much clearer separation between digit classes than the standard RBF kernel.

Figure 2: Latent space visualizations for the Oil Flow (top) and USPS (bottom) datasets. The MLP kernel (bottom row) shows significantly better class separation for digits compared to the RBF kernel.

Uncertainty Propagation: Airline Passengers

In time-series "free simulation" tasks, where the model must predict multiple steps ahead by propagating its own uncertainty, the UT approach proved highly effective. It achieved predictive performance (measured by Negative Log Predictive Density) comparable to GH quadrature but with a 170-fold speedup, capturing periodic trends that low-sample Monte Carlo methods failed to resolve accurately.

Conclusion

The Unscented Gaussian Process Latent Variable Model provides a robust "sweet spot" between the exponential cost of quadrature and the stochastic instability of Monte Carlo methods. By maintaining a deterministic framework with linear scaling, it unlocks the use of complex kernels for high-dimensional latent variable modeling. Future work includes extending this framework to Deep Gaussian Processes and exploring non-Gaussian likelihoods within the same variational inference scheme.

Summary updated

Abstract

Gaussian Process Latent Variable Model (GPLVM) is a flexible framework to handle uncertain inputs in Gaussian Processes (GPs) and incorporate GPs as components of larger graphical models. Nonetheless, the standard GPLVM variational inference approach is tractable only for a narrow family of kernel functions. The most popular implementations of GPLVM circumvent this limitation using quadrature methods, which may become a computational bottleneck even for relatively low dimensions. For instance, the widely employed Gauss-Hermite quadrature has exponential complexity on the number of dimensions. In this work, we propose using the unscented transformation instead. Overall, this method presents comparable, if not better, performance than offthe-shelf solutions to GPLVM and its computational complexity scales only linearly on dimension. In contrast to Monte Carlo methods, our approach is deterministic and works well with quasi-Newton methods, such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. We illustrate the applicability of our method with experiments on dimensionality reduction and multistep-ahead prediction with uncertainty propagation.
BibTeX
@misc{souza2019unscented,
  author = {Daniel Augusto R. M. A. de Souza and César Lincoln C. Mattos and João Paulo Pordeus Gomes},
  title = {{Unscented Gaussian Process Latent Variable Model: learning from uncertain inputs with intractable kernels}},
  year = {2019},
  eprint = {1907.01867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.01867},
}
Figure