TL;DR
This paper introduces and evaluates fast, accurate approximations for Bayesian leave-one-out (LOO) cross-validation in Gaussian Latent Variable Models (GLVMs). The key takeaway is that approximations based on the "cavity distribution" (a Gaussian approximation to the marginal posterior after removing one data point) offer the best balance of accuracy and computational efficiency.
What Problem Does This Paper Address?
When building Bayesian models, especially complex ones like Gaussian Latent Variable Models (GLVMs), we need to assess how well they predict unseen data. This is crucial for model selection and understanding model performance. A standard method for this is Leave-One-Out (LOO) cross-validation, where we iteratively remove one data point, train the model on the rest, and predict the removed point. However, performing this "brute force" for every data point can be computationally prohibitive, especially with large datasets. This paper addresses the challenge of developing and comparing fast approximations to this exact LOO procedure for GLVMs, specifically when the integration over latent variables is handled by Laplace approximation or Expectation Propagation (EP).
Methods
The authors investigate several methods for approximating LOO cross-validation. The core idea is to leverage the posterior approximation obtained from the full dataset to quickly estimate the LOO predictive performance.
- Laplace Approximation (LA): Approximates the posterior distribution of latent variables with a Gaussian centered at the mode.
- Expectation Propagation (EP): An iterative method that approximates the posterior with a Gaussian by matching moments of the true posterior with a simpler approximating distribution.
- Cavity Distribution: A key concept introduced is the "cavity distribution," which is essentially the approximate posterior distribution of the latent variables after removing one data point. The paper shows that approximating the LOO marginal with this cavity distribution, often derived from LA or EP, leads to accurate results.
- Global vs. Local Corrections: They compare methods that use local information (e.g., only the removed data point) versus global information (e.g., considering the effect on all latent variables) to improve the approximations.
- Quadrature and Importance Sampling: For cases where analytical solutions are not possible, they explore numerical integration (quadrature) and importance sampling techniques.
Datasets
The paper evaluates their methods on several real-world datasets:
- Classification Datasets: Ripley, Australian, Ionosphere, Sonar. These were chosen to explore scenarios with skewed posteriors and varying numbers of covariates.
- Survival Data: Leukemia (with censoring).
- Regression Dataset: Boston (using a Student's t likelihood). This dataset was specifically chosen to test performance with non-log-concave likelihoods, which can lead to multimodal posteriors.
The paper does not introduce new benchmark datasets.
Key Insights and Ideas
- Cavity Distribution is Key: The most significant finding is that approximations based on the Gaussian "cavity distribution" (derived from LA or EP) provide the most accurate and reliable LOO estimates among the fast methods. This suggests that accurately approximating the marginal posterior after removing a data point is more important than complex global corrections for achieving good LOO performance.
- LA and EP are Competitive: Both Laplace approximation and Expectation Propagation, when used to derive the cavity distribution, yield good results. EP is generally considered more accurate but computationally more intensive than LA.
- Global Corrections Can Be Unstable: While global corrections aim to improve accuracy by accounting for more complex posterior shapes, they can introduce instability, especially in cases with many parameters or when the posterior is far from Gaussian. The paper shows that these complex corrections don't always translate to substantial improvements in LOO accuracy.
- Computational Efficiency is Achievable: The paper demonstrates that accurate LOO estimates can be obtained with minimal additional computational cost after the initial posterior approximation is formed, making it practical for large datasets.
- Model Flexibility Matters: As model flexibility increases (e.g., shorter length scales in Gaussian Processes), the gap between the full posterior and the LOO posterior widens, making approximations more challenging. LA-LOO and EP-LOO remain robust even in these scenarios, while other methods struggle.
Key Results
- LA-LOO and EP-LOO Excel: The paper's empirical results consistently show that LA-LOO and EP-LOO (using the cavity distribution) provide the best accuracy for LOO cross-validation across various datasets and model complexities.
- Quadrature Methods Perform Well (with caveats): Quadrature-based methods (Q-LOO) using improved marginal approximations (like LA-CM2) also show good results, but can sometimes fail or be less accurate than cavity-based methods, especially with non-Gaussian posteriors.
- WAIC's Performance: WAIC (Widely Applicable Information Criterion) generally performs worse than the best LOO approximations, particularly in cases of high model flexibility or when the posterior is far from Gaussian.
- Computational Cost: EP-LOO has zero additional computational cost after EP is performed. LA-LOO has negligible cost, requiring only a few quadrature integrals. More complex methods like those involving global corrections or extensive quadrature can significantly increase computation time.
Limitations and Future Work
- Focus on GLVMs: The presented methods are primarily developed for Gaussian Latent Variable Models. Extending these to other model classes (e.g., generalized linear models, deep generative models) is an area for future work.
- Non-Gaussian Likelihoods: While the paper considers non-Gaussian likelihoods, the focus is on approximations that result in Gaussian posteriors. Further research into non-Gaussian approximations for LOO could be beneficial.
- Model Selection Bias: The authors note that while LOO is useful for model comparison, it can be biased when used to select a single model from a large set of candidates. This is an area of ongoing research.
- Hyperparameter Integration: The paper explores integrating over hyperparameters, but the computational cost can be significant. More efficient methods for this integration in the context of LOO are still valuable.
Summary updated