Abstract
Sparse Gaussian process methods that use inducing variables require the selection of the inducing inputs and the kernel hyperparameters. We introduce a variational formulation for sparse approximations that allows for the joint optimization of inducing inputs and kernel hyperparameters by minimizing the Kullback-Leibler divergence between the approximation and the true posterior. This approach provides a principled way to avoid overfitting and rigorously approximate the true GP posterior.
Introduction
Gaussian Processes (GPs) are the "gold standard" for Bayesian non-parametric regression, offering elegant uncertainty estimates and flexible modeling. However, they have a notorious Achilles' heel: computational complexity. Standard GPs scale at $O(n^3)$, making them unusable for datasets with more than a few thousand points.
To handle large datasets, researchers use "sparse" GPs. These methods approximate the full GP using a small set of $m$ "inducing variables" (or pseudo-inputs), reducing complexity to $O(nm^2)$. The challenge lies in choosing where to put these $m$ points and how to set the kernel hyperparameters. Previous state-of-the-art methods, like the Sparse Pseudo-input GP (SPGP), treated these inducing locations as additional kernel hyperparameters. While flexible, this often leads to overfitting: the model can "cheat" by placing inducing points in a way that interpolates noise rather than capturing the underlying signal.
Variational Learning Framework
This paper introduces a variational framework (VAR) for sparse GPs that treats the selection of inducing variables as a variational inference problem rather than a standard optimization task. The core idea is to define a variational distribution over the latent function values and minimize the Kullback-Leibler (KL) divergence between this approximation and the exact GP posterior.
Key technical contributions include:
- Inducing Inputs as Variational Parameters: Instead of being model hyperparameters, the inducing locations $X_m$ are parameters of the variational distribution.
- A True Lower Bound: The author derives a new objective function ($F_V$) that is a mathematically guaranteed lower bound of the true log marginal likelihood.
- The Regularization Trace Term: The objective function includes a unique term: $-rac{1}{2\sigma^2} Tr(\tilde{K})$. This term represents the total variance of the training data that is not captured by the inducing variables. Maximizing the bound naturally forces the inducing points to "cover" the training data effectively.
Experimental Evaluation
The method was evaluated on several standard regression benchmarks, including a 1D toy dataset, Boston Housing, KIN40K, and SARCOS.
Overfitting Resistance
On a 20-point toy dataset, the VAR method correctly identified the smooth underlying function, while SPGP and Projected Process (PP) methods produced "wiggly" curves that overfit the noise. Because the inducing inputs are variational parameters, the model cannot overfit by simply adding more of them; it only improves the approximation of the true GP.
Large-Scale Performance
On larger datasets like KIN40K (10,000 points) and SARCOS (44,484 points), the VAR method consistently matched or outperformed other sparse methods in terms of Standardized Mean Squared Error (SMSE) and Standardized Negative Log Probability (SNLP). Unlike previous methods where adding inducing points could occasionally decrease the marginal likelihood, in the VAR framework, adding a point is guaranteed to never decrease the lower bound.
Conclusion
The variational formulation provides a principled way to learn inducing variables in sparse GPs. By treating inducing locations as variational parameters, the method offers natural regularization and a monotonic improvement in the approximation quality as more inducing points are added. This framework resolves long-standing issues of overfitting in sparse GP approximations and provides a robust tool for large-scale Bayesian regression.
Summary updated