Abstract
The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when making predictions. This integration is often performed using Markov chain Monte Carlo (MCMC) sampling. However, with non-Gaussian observations standard hyperparameter sampling approaches require careful tuning and may converge slowly. In this paper we present a slice sampling approach that requires little tuning while mixing well in both strong- and weak-data regimes.
Introduction: The Coupling Trap
Gaussian Processes (GPs) are fundamental tools in Bayesian modeling for defining dependencies between variables. Their behavior is governed by hyperparameters $\theta$, such as lengthscale and signal variance. To perform full Bayesian inference, we must integrate over these hyperparameters, typically using Markov chain Monte Carlo (MCMC).
However, a significant challenge arises: the latent variables $\mathbf{f}$ and the hyperparameters $\theta$ are strongly coupled. In standard MCMC schemes that alternate between updating $\mathbf{f}$ and $\theta$, the conditional posterior $p(\theta | \mathbf{f})$ becomes extremely peaked. If one attempts to update the lengthscale while keeping $\mathbf{f}$ fixed, the new $\theta$ is likely to be highly improbable, causing the sampler to stall. This "locking" effect is particularly severe in non-Gaussian models, such as those used for classification or count data, where the latent variables cannot be analytically integrated out.
Methodology: Decoupling via Surrogate Data
To address this coupling, we introduce a "surrogate data" model. The core idea is to introduce an auxiliary Gaussian variable $\mathbf{g}$ that acts as a "noisy" version of the latent variables $\mathbf{f}$. By conditioning on this surrogate data rather than the raw latents, we create a smoother pseudo-posterior.
This representation allows the hyperparameters to change while the latent variables "float" and adapt to the new covariance structure. Specifically, we use slice sampling for the hyperparameter updates. Unlike Metropolis-Hastings, slice sampling automatically adjusts its scale, making the algorithm robust and "plug-and-play" without the need for manual step-size tuning. A key innovation is the surr-site heuristic, which automatically sets the surrogate noise by matching the curvature of the likelihood via a Laplace approximation, ensuring the model adapts to the specific problem geometry.
Experimental Evaluation
We evaluated the surrogate slice sampler across four distinct data regimes to test its versatility:
- Binary Classification (Ionosphere): 200 training points with 34 dimensions.
- Gaussian Regression (Synthetic): 200 points in 10 dimensions, used as a sanity check against the ideal analytical solution.
- Cox Process (Mining Disasters): A 1D temporal point process tracking 191 events over time.
- Cox Process (Redwood Trees): A 2D spatial point process tracking 195 tree locations.
Results and Discussion
The performance was measured in Effective Samples per Second (ESS/s), which accounts for both mixing quality and computational cost. The surrogate methods (surr-site) significantly outperformed standard baselines on the Ionosphere and Redwood datasets. In the synthetic Gaussian case, the method achieved efficiency comparable to the ideal reparameterization, demonstrating that the auxiliary variable introduces minimal overhead.
A critical insight from our study is that common "whitening" tricks (transforming variables to be independent under the prior) are not universal solutions. While whitening helps when data is sparse ("weak data"), it actually exacerbates the coupling problem in data-rich regimes. The surrogate model acts as a robust bridge, performing well regardless of data density.
Conclusion
The surrogate slice sampler provides a robust, tuning-free method for hyperparameter inference in latent Gaussian models. While it maintains the $O(N^3)$ complexity typical of GPs, its consistency across different data regimes makes it a valuable tool for practitioners. Future work could explore combining these surrogate representations with gradient-based methods like Hamiltonian Monte Carlo (HMC) for even higher-dimensional hyperparameter spaces.
Summary updated