Abstract
Gaussian Process Latent Variable Model (GPLVM) is a flexible framework to handle uncertain inputs in Gaussian Processes (GPs) and incorporate GPs as components of larger graphical models. Nonetheless, the standard GPLVM variational inference approach is tractable only for a narrow family of kernel functions. The most popular implementations of GPLVM circumvent this limitation using quadrature methods, which may become a computational bottleneck even for relatively low dimensions. For instance, the widely employed Gauss-Hermite quadrature has exponential complexity on the number of dimensions. In this work, we propose using the unscented transformation instead. Overall, this method presents comparable, if not better, performance than off-the-shelf solutions to GPLVM and its computational complexity scales only linearly on dimension. In contrast to Monte Carlo methods, our approach is deterministic and works well with quasi-Newton methods, such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. We illustrate the applicability of our method with experiments on dimensionality reduction and multistep-ahead prediction with uncertainty propagation.
The Bottleneck of Bayesian GPLVMs
Gaussian Process Latent Variable Models (GPLVMs) are powerful tools for dimensionality reduction and modeling systems with uncertain inputs. However, the Bayesian version of the model—which accounts for uncertainty in the latent space—faces a significant mathematical hurdle. To perform variational inference, one must solve complex integrals known as $\Psi$-statistics, which involve convolving a kernel function with a Gaussian distribution.
Historically, these integrals were only analytically tractable for a few simple kernels, most notably the Radial Basis Function (RBF). While RBF is a standard choice, its extreme smoothness often limits its ability to capture complex patterns or extrapolate effectively. To use more expressive kernels like Matérn or Multilayer Perceptron (MLP), researchers typically resort to numerical approximations:
- Gauss-Hermite (GH) Quadrature: Highly accurate but suffers from exponential complexity $O(H^D)$ with respect to the latent dimension $D$.
- Monte Carlo (MC) Sampling: Flexible but stochastic. The resulting noise in the objective function prevents the use of efficient, deterministic second-order optimizers like BFGS.
Scaling with the Unscented Transformation
This work proposes using the Unscented Transformation (UT) to approximate $\Psi$-statistics. Originally developed for control theory (e.g., the Unscented Kalman Filter), UT selects a small, deterministic set of "sigma points" that capture the mean and covariance of the input distribution.
For a $D$-dimensional latent space, UT requires only $2D+1$ evaluations, providing linear scaling compared to the exponential cost of GH quadrature. As shown in the figure below, the relative time spent on UT compared to GH drops precipitously as the input dimension increases.
Beyond efficiency, UT is deterministic. This stability allows for the use of quasi-Newton optimization methods like BFGS, which often converge faster and more reliably than the first-order stochastic gradient methods (like Adam) required by Monte Carlo approaches.
Experimental Evaluation
The authors evaluated the UT-based GPLVM across several benchmarks, demonstrating its ability to unlock expressive kernels that were previously computationally prohibitive.
Dimensionality Reduction: Oil Flow and USPS
On the Oil Flow dataset, the UT approach with a Matérn 3/2 kernel achieved 100% accuracy in 1-Nearest Neighbor classification, outperforming the standard analytic RBF solution. In the USPS digit dataset, the MLP kernel (enabled by UT) provided significantly better class separation in the latent space than the RBF kernel, which tended to cluster different digits together.
Time-Series Prediction: Airline Passengers
In multi-step-ahead prediction tasks using the Airline Passenger dataset, the UT-based model successfully propagated uncertainty through time. It achieved results comparable to GH quadrature but with a massive 170-fold reduction in the number of function evaluations required for a 7-dimensional latent space.
Conclusion
By replacing exponential-cost quadrature or noisy sampling with the Unscented Transformation, this method provides a scalable, deterministic framework for Bayesian GPLVMs. This "efficiency-accuracy" sweet spot allows researchers to move beyond the RBF kernel, utilizing more expressive kernels while maintaining computational feasibility in higher-dimensional latent spaces.
Summary updated