Bayesian leave-one-out cross-validation approximations for Gaussian\n latent variable models

Aki Vehtari, Tommi Mononen, Ville Tolvanen, Tuomas Sivula, Ole Winther

Abstract

Estimating the predictive performance of Bayesian models often relies on cross-validation. For Gaussian latent variable models (GLVMs), where latent values must be integrated out, this process is computationally demanding. This paper investigates several fast approximations for Bayesian leave-one-out (LOO) cross-validation using the Laplace method and expectation propagation (EP). By comparing these fast methods against brute-force LOO and ground-truth Markov chain Monte Carlo (MCMC) simulations, the authors demonstrate that Gaussian approximations to the LOO marginal distribution—specifically the cavity distribution—provide highly accurate and computationally efficient results.

Introduction

In Bayesian statistics, assessing a model's ability to generalize to unseen data is a critical step in the modeling pipeline. Leave-one-out cross-validation (LOO-CV) is widely regarded as the gold standard for this task. However, its implementation requires re-fitting the model $n$ times, where $n$ is the number of data points. For complex models such as Gaussian Processes (GPs) or other Gaussian Latent Variable Models (GLVMs), this brute-force approach is often computationally prohibitive. While information criteria like WAIC offer faster alternatives, they can be unreliable in flexible models or when specific data points exert significant influence. This paper explores more robust, model-specific approximations that leverage the structure of GLVMs to provide near-instantaneous LOO estimates.

The Problem: Computational Bottlenecks in GLVMs

GLVMs are a broad class of models where observations are linked to latent variables through a likelihood function, and the latent variables themselves follow a Gaussian prior. The primary challenge lies in the marginalization of these latent variables. In a full Bayesian treatment, this integration is often handled via MCMC, which is slow. Approximations like the Laplace method (LA) and Expectation Propagation (EP) are commonly used to speed up the initial model fit. The authors' goal is to extend these approximations to the LOO-CV context, allowing practitioners to evaluate model performance without the need for repeated re-fitting.

Approximation Methods

The authors focus on two main frameworks: the Laplace Approximation and Expectation Propagation. The core idea is to approximate the "cavity distribution"—the distribution of a latent variable $f_i$ given all data except the $i$-th observation ($y_{-i}$).

Summary of Laplace and Expectation Propagation based approximations for the LOO marginal distribution.

As shown in the table above, several variants of these approximations exist. The simplest approach (LA-G or EP-G) uses the Gaussian marginal from the joint distribution. More sophisticated methods introduce "local" or "global" improvements. For instance, LA-L and EP-L use tilted distributions to better capture the local behavior of the likelihood. The most advanced methods, such as LA-TK and EP-FULL, apply global corrections to the marginal distribution, often resulting in higher accuracy at the cost of slight complexity.

A key theoretical contribution is the application of Linear Response Theory to the Laplace method. This allows for the derivation of LOO estimates analytically, transforming a repetitive training task into a single-pass calculation. Additionally, to handle likelihoods with heavy tails (e.g., Student’s $t$ distributions), the authors introduce Truncated Weights Quadrature (TQ-LOO) to stabilize numerical integration.

Experimental Evaluation

The methods were tested across various datasets, including classification (Ripley, Australian, Ionosphere, Sonar), survival analysis (Leukemia), and robust regression (Boston housing). These datasets represent a range of challenges, from high-dimensional feature spaces to non-Gaussian likelihoods.

Key Insights and Results

  • The Superiority of Cavity Distributions: The Gaussian approximation of the LOO marginal distribution (the cavity distribution) consistently outperformed general-purpose metrics like WAIC. It provided a more stable and accurate estimate of the predictive density.
  • Failure Modes of WAIC: WAIC was found to be particularly weak in highly flexible models, such as GPs with short length scales, where it exhibited high bias and variance.
  • Computational Efficiency: The speedups achieved are dramatic. In the Leukemia dataset, while brute-force EP required over an hour (~3,700 seconds), the EP-LOO approximation provided results in just 0.005 seconds—a speedup of several orders of magnitude with negligible loss in accuracy.
  • Stability: The proposed "truncated weights" approach makes these fast methods much more robust for "messy" real-world data where standard Importance Sampling might fail.

Conclusion and Future Work

This research demonstrates that for Gaussian Latent Variable Models, fast LOO-CV approximations are not only possible but highly reliable. By utilizing cavity distributions and linear response theory, researchers can obtain accurate performance metrics at a fraction of the computational cost of traditional methods.

Future work remains in addressing hyperparameter sensitivity. While the methods work excellently for latent variables, they still rely on Importance Sampling for top-level hyperparameters. Extending these techniques to non-factorizing likelihoods, such as those found in multi-class classification, also presents an interesting avenue for further study.

Summary updated

Abstract

The future predictive performance of a Bayesian model can be estimated using\nBayesian cross-validation. In this article, we consider Gaussian latent\nvariable models where the integration over the latent values is approximated\nusing the Laplace method or expectation propagation (EP). We study the\nproperties of several Bayesian leave-one-out (LOO) cross-validation\napproximations that in most cases can be computed with a small additional cost\nafter forming the posterior approximation given the full data. Our main\nobjective is to assess the accuracy of the approximative LOO cross-validation\nestimators. That is, for each method (Laplace and EP) we compare the\napproximate fast computation with the exact brute force LOO computation.\nSecondarily, we evaluate the accuracy of the Laplace and EP approximations\nthemselves against a ground truth established through extensive Markov chain\nMonte Carlo simulation. Our empirical results show that the approach based upon\na Gaussian approximation to the LOO marginal distribution (the so-called cavity\ndistribution) gives the most accurate and reliable results among the fast\nmethods.\n
BibTeX
@article{vehtari2016bayesian,
  author = {Aki Vehtari and Tommi Mononen and Ville Tolvanen and Tuomas Sivula and Ole Winther},
  title = {{Bayesian leave-one-out cross-validation approximations for Gaussian\textbackslash\{\}n latent variable models}},
  journal = {J. Mach. Learn. Res.},
  year = {2016},
  url = {https://openreview.net/forum?id=t9lpdJNUWyE},
}

Loading related content...

Figure