Slice sampling covariance hyperparameters of latent Gaussian models

Iain Murray, Ryan Prescott Adams

This paper, presented by Iain Murray and Ryan Prescott Adams, introduces a robust Markov chain Monte Carlo (MCMC) method for sampling the hyperparameters of Latent Gaussian Models (LGMs). By introducing a "surrogate data" auxiliary variable, the authors decouple the strong dependencies between latent variables and hyperparameters. This results in a slice sampler that requires minimal tuning and performs consistently across various data regimes—providing a "tuning-light" alternative to Hamiltonian Monte Carlo that works reliably whether your data is sparse and weak or dense and informative.

The Problem: Coupling in LGMs

Latent Gaussian Models (LGMs)—which include Gaussian Processes (GPs) and generalized linear mixed models—are workhorses of modern statistics. However, performing Bayesian inference on them is notoriously difficult due to a core issue: coupling.

In an LGM, you have latent variables $\mathbf{f}$ (the "hidden" Gaussian values) and hyperparameters $\theta$ (which control the shape and scale of that Gaussian distribution, such as the lengthscale). If you try to update $\theta$ while keeping $\mathbf{f}$ fixed, you find that $\mathbf{f}$ is only "plausible" for a very narrow range of $\theta$. Conversely, if you change $\theta$ significantly, the current $\mathbf{f}$ becomes extremely unlikely under the new prior.

Prior draws and the resulting narrow posterior for lengthscale, illustrating the tight coupling between latent variables and hyperparameters.

Because of this tight coupling, standard MCMC methods often get "stuck," vibrating in a tiny region of the parameter space and taking forever to explore the full posterior. While "whitening" the prior (transforming variables to be independent under the prior) helps when data is weak, it fails when the data is "strong" (informative), as the likelihood then creates new, tight couplings.

The Solution: Surrogate Data Models and Slice Sampling

To break this crippling correlation, the authors introduce a Surrogate Data Model. The key idea is to augment the model with an auxiliary variable $\mathbf{g}$, which acts as a "noisy" version of the latent variables $\mathbf{f}$.

  1. The Auxiliary Step: Instead of conditioning directly on $\mathbf{f}$, the algorithm conditions on a "whitened" version of $\mathbf{f}$ relative to a surrogate posterior.
  2. Surrogate Covariance ($S_\theta$): The "noise" level of this surrogate data is crucial. The authors provide an automated way to set this by matching the "site-specific" variance of the likelihood. This allows the algorithm to adapt to how much the data is actually constraining each latent variable.
  3. Slice Sampling: Rather than using Metropolis-Hastings (which requires a carefully tuned proposal distribution), they use Slice Sampling. This is a "tuning-free" MCMC variant that automatically adjusts its scale to find the "slice" of the distribution with high probability.
Comparison of proposals: the surrogate data proposal remains much closer to the current state than the whitened prior proposal.

By combining these techniques, they create a joint update for $\theta$ and $\mathbf{f}$ that "wiggles" the latent variables in a way that remains consistent with both the new hyperparameters and the observed data. The surrogate data acts as a bridge: it provides a "soft" constraint that guides the latent variables to stay in a plausible region during a hyperparameter move, avoiding the rigidity of fixing $\mathbf{f}$ and the looseness of ignoring the data.

Experimental Validation

The authors evaluated their method on four distinct problems:

  • Binary Classification (Ionosphere): A real-world dataset with 200 samples and 34 dimensions, using a logistic likelihood.
  • Gaussian Regression (Synthetic): A 10-dimensional synthetic problem designed to test the method against an "ideal" case where the latent variables can be marginalized out analytically.
  • 1D Cox Process (Mining Disasters): A point-process model tracking coal mining disasters over time.
  • 2D Cox Process (Redwood Trees): A spatial point-process model tracking the locations of redwood trees, involving a large number of spatial bins.

Across the board, the surrogate data method (surr-site) proved to be a top performer. It consistently produced more "effective samples" per second and per covariance construction than standard baselines. In the Ionosphere classification task, the surrogate methods worked significantly better than the baselines. Furthermore, in the synthetic regression case, the method performed nearly as well as the "ideal" reparameterization, proving that the "random centering" of the surrogate model doesn't sacrifice much efficiency.

Key Insights and Future Work

  • Tuning is the Enemy: Methods like Hamiltonian Monte Carlo (HMC) are powerful but require tuning step sizes and mass matrices. This slice sampler is designed to be "plug-and-play" for researchers who want robust inference without the headache of manual calibration.
  • Handling Flat Likelihoods: The paper shows that Taylor-expansion-based methods fail when the likelihood is non-Gaussian and "flat" (e.g., in classification where some points are perfectly predicted). The surr-site method handles these edge cases gracefully.

Limitations: The method still requires $O(N^3)$ matrix operations (Cholesky decompositions) for each hyperparameter proposal. While this is standard for GPs, it remains the primary bottleneck for scaling to massive datasets. Additionally, the current implementation updates hyperparameters one by one (axis-aligned). If hyperparameters are themselves strongly correlated, more sophisticated reparameterizations or joint moves might be needed. Ultimately, the authors suggest this representation could be combined with other MCMC techniques, like HMC, to potentially get the "best of both worlds" in the future.

Summary updated

Abstract

The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when making predictions. This integration is often performed using Markov chain Monte Carlo (MCMC) sampling. However, with non-Gaussian observations standard hyperparameter sampling approaches require careful tuning and may converge slowly. In this paper we present a slice sampling approach that requires little tuning while mixing well in both strong- and weak-data regimes.
BibTeX
@inproceedings{murray2010slice,
  author = {Iain Murray and Ryan Prescott Adams},
  title = {{Slice sampling covariance hyperparameters of latent Gaussian models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=Sy-v2D-_WH},
}
Figure