Learning GPLVM with arbitrary kernels using the unscented transformation

Daniel Augusto de Souza, Diego Mesquita, César Mattos, João P. P. Gomes

Abstract

Gaussian Process Latent Variable Model (GPLVM) is a flexible framework to handle uncertain inputs in Gaussian Processes (GPs) and incorporate GPs as components of larger graphical models. Nonetheless, the standard GPLVM variational inference approach is tractable only for a narrow family of kernel functions. The most popular implementations of GPLVM circumvent this limitation using quadrature methods, which may become a computational bottleneck even for relatively low dimensions. For instance, the widely employed Gauss-Hermite quadrature has exponential complexity on the number of dimensions. In this work, we propose using the unscented transformation instead. Overall, this method presents comparable, if not better, performance than off-the-shelf solutions to GPLVM and its computational complexity scales only linearly on dimension. In contrast to Monte Carlo methods, our approach is deterministic and works well with quasi-Newton methods, such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. We illustrate the applicability of our method with experiments on dimensionality reduction and multistep-ahead prediction with uncertainty propagation.

The Bottleneck of Bayesian GPLVMs

Gaussian Process Latent Variable Models (GPLVMs) are powerful tools for dimensionality reduction and modeling systems with uncertain inputs. However, the Bayesian version of the model—which accounts for uncertainty in the latent space—faces a significant mathematical hurdle. To perform variational inference, one must solve complex integrals known as $\Psi$-statistics, which involve convolving a kernel function with a Gaussian distribution.

Historically, these integrals were only analytically tractable for a few simple kernels, most notably the Radial Basis Function (RBF). While RBF is a standard choice, its extreme smoothness often limits its ability to capture complex patterns or extrapolate effectively. To use more expressive kernels like Matérn or Multilayer Perceptron (MLP), researchers typically resort to numerical approximations:

  1. Gauss-Hermite (GH) Quadrature: Highly accurate but suffers from exponential complexity $O(H^D)$ with respect to the latent dimension $D$.
  2. Monte Carlo (MC) Sampling: Flexible but stochastic. The resulting noise in the objective function prevents the use of efficient, deterministic second-order optimizers like BFGS.

Scaling with the Unscented Transformation

This work proposes using the Unscented Transformation (UT) to approximate $\Psi$-statistics. Originally developed for control theory (e.g., the Unscented Kalman Filter), UT selects a small, deterministic set of "sigma points" that capture the mean and covariance of the input distribution.

For a $D$-dimensional latent space, UT requires only $2D+1$ evaluations, providing linear scaling compared to the exponential cost of GH quadrature. As shown in the figure below, the relative time spent on UT compared to GH drops precipitously as the input dimension increases.

Figure 1: Relative time spent of UT compared to other methods as input dimension increases. Note the exponential advantage over Gauss-Hermite (GH) quadrature.

Beyond efficiency, UT is deterministic. This stability allows for the use of quasi-Newton optimization methods like BFGS, which often converge faster and more reliably than the first-order stochastic gradient methods (like Adam) required by Monte Carlo approaches.

Experimental Evaluation

The authors evaluated the UT-based GPLVM across several benchmarks, demonstrating its ability to unlock expressive kernels that were previously computationally prohibitive.

Dimensionality Reduction: Oil Flow and USPS

On the Oil Flow dataset, the UT approach with a Matérn 3/2 kernel achieved 100% accuracy in 1-Nearest Neighbor classification, outperforming the standard analytic RBF solution. In the USPS digit dataset, the MLP kernel (enabled by UT) provided significantly better class separation in the latent space than the RBF kernel, which tended to cluster different digits together.

Figure 2: Latent space visualizations for Oil Flow (top) and USPS (bottom) datasets. UT enables the use of Matérn and MLP kernels, which provide better class separation than the standard analytic RBF.

Time-Series Prediction: Airline Passengers

In multi-step-ahead prediction tasks using the Airline Passenger dataset, the UT-based model successfully propagated uncertainty through time. It achieved results comparable to GH quadrature but with a massive 170-fold reduction in the number of function evaluations required for a 7-dimensional latent space.

Conclusion

By replacing exponential-cost quadrature or noisy sampling with the Unscented Transformation, this method provides a scalable, deterministic framework for Bayesian GPLVMs. This "efficiency-accuracy" sweet spot allows researchers to move beyond the RBF kernel, utilizing more expressive kernels while maintaining computational feasibility in higher-dimensional latent spaces.

Summary updated

Abstract

Gaussian Process Latent Variable Model (GPLVM) is a flexible framework to handle uncertain inputs in Gaussian Processes (GPs) and incorporate GPs as components of larger graphical models. Nonetheless, the standard GPLVM variational inference approach is tractable only for a narrow family of kernel functions. The most popular implementations of GPLVM circumvent this limitation using quadrature methods, which may become a computational bottleneck even for relatively low dimensions. For instance, the widely employed Gauss-Hermite quadrature has exponential complexity on the number of dimensions. In this work, we propose using the unscented transformation instead. Overall, this method presents comparable, if not better, performance than offthe-shelf solutions to GPLVM and its computational complexity scales only linearly on dimension. In contrast to Monte Carlo methods, our approach is deterministic and works well with quasi-Newton methods, such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. We illustrate the applicability of our method with experiments on dimensionality reduction and multistep-ahead prediction with uncertainty propagation.
BibTeX
@misc{souza2019learning,
  author = {Daniel Augusto de Souza and Diego Mesquita and César Mattos and João P. P. Gomes},
  title = {{Learning GPLVM with arbitrary kernels using the unscented transformation}},
  year = {2019},
  eprint = {1907.01867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.01867},
}

Loading related content...

Figure