Flexible Variational Bayes Based on a Copula of a Mixture

David Gunawan, Robert Kohn, David J. NottJ. Comput. Graph. Stat 2024 · DOI

TL;DR

This paper introduces a novel variational Bayes (VB) method that uses a mixture of Gaussian copulas to flexibly approximate complex posterior distributions, outperforming existing methods in accuracy and efficiency. The approach is particularly effective for multimodal, skewed, and heavy-tailed posteriors, demonstrating its utility in challenging Bayesian deep learning models.

What Problem Does This Paper Address?

Bayesian inference often involves estimating posterior distributions, which can be computationally intractable for complex models. Variational Bayes (VB) offers a computationally cheaper alternative by approximating the posterior with a simpler, tractable distribution. However, standard VB methods often struggle with posteriors that are multimodal, skewed, or heavy-tailed, as they typically rely on Gaussian approximations. This limitation hinders accurate inference and prediction in many real-world applications, especially in areas like deep learning where such complex posteriors are common. The paper aims to bridge this gap by developing a more flexible VB approximation that can capture these challenging posterior shapes.

Methods

The core innovation is a flexible variational approximation based on a copula of a mixture. The key idea is to first transform each parameter's marginal distribution using a flexible transformation (like Yeo-Johnson) to make it more amenable to modeling. Then, a mixture of Gaussian distributions is used to model the joint distribution of these transformed parameters. This "copula of a mixture" approach allows for richer posterior approximations than standard Gaussian or even Gaussian copula methods.

The paper also introduces a boosting variational inference algorithm that adds mixture components one at a time, refining the approximation iteratively. This boosting approach is made efficient by employing natural gradient methods and variance reduction techniques. Natural gradients exploit the information geometry of the variational distribution for faster convergence, while variance reduction techniques help stabilize gradient estimates, especially in high-dimensional settings. The authors also propose methods for initializing new mixture components to ensure stable and efficient learning.

Datasets

The paper evaluates its proposed methods on several simulated and real-world datasets:

  • Simulated Datasets:
    • A 100-dimensional multivariate t-copula for skewed, heavy-tailed, and high-dimensional distributions.
  • A 100-dimensional mixture of normals for multimodal and high-dimensional distributions.
  • A Gaussian target distribution to show the method does not overfit.
  • Real-world Datasets:
    • Bayesian Logistic Regression: Spam, krkp, ionosphere, and mushroom datasets (using the first 50 observations and 7 covariates for spam, and all covariates for others).
  • Bayesian Deep Neural Network Regression: Auto and abalone datasets, with various neural network structures (e.g., (8,5,5,1), (8,10,10,1), (8,20,20,1) for auto; (9,5,5,1), (9,10,10,1), (9,20,20,1) for abalone).

Key Insights and Ideas

  • Transforming Marginals is Key: The paper highlights that transforming individual parameter marginals before modeling their joint distribution (via a copula) is a powerful way to achieve flexibility, rather than relying solely on a complex multivariate distribution.
  • Boosting for Complexity: Adding mixture components incrementally (boosting) is an effective strategy for building increasingly complex and accurate posterior approximations, allowing for a tunable trade-off between accuracy and computational cost.
  • Natural Gradients Accelerate Learning: The use of natural gradients significantly speeds up the optimization process for variational inference, especially in high-dimensional models, by accounting for the geometry of the parameter space.
  • Copula of a Mixture is Universally Approximating: The authors suggest that their proposed method, under reasonable assumptions, can act as a universal approximator of multivariate distributions, a property not guaranteed by mixtures of Gaussian copulas alone.
  • HMC Struggles with Complex Posteriors: The experiments demonstrate that traditional MCMC methods like Hamiltonian Monte Carlo (HMC) can be inefficient and struggle to converge for the complex, high-dimensional, multimodal, and heavy-tailed posteriors that the proposed VB method handles well.

Key Results

  • Improved Accuracy: The proposed "copula of a mixture" variational approximation (CMGVA) consistently outperforms simpler Gaussian and Gaussian copula approximations in capturing the shape of multimodal, skewed, and heavy-tailed posteriors, as evidenced by higher lower bounds and better density estimates.
  • Efficiency Gains: The CMGVA is significantly faster than HMC for complex models, with speedups of 2-4 times reported for some examples. The use of natural gradients further enhances optimization speed and stability.
  • Deep Learning Performance: The method shows strong performance in Bayesian deep neural network regression, achieving better lower bounds and predictive scores compared to simpler VB methods and planar flows.
  • Component Addition Matters: Adding a few mixture components generally improves the approximation quality, but diminishing returns are observed after a certain number of components.

Limitations and Future Work

  • Computational Cost: While more efficient than HMC, fitting complex mixture models can still be computationally intensive, especially with a large number of components.
  • Initialization Sensitivity: The performance of boosting methods can be sensitive to the initialization of new components, although the paper proposes effective initialization strategies.
  • Extension to Other Copulas: The authors note that it is straightforward to use other copula-based approximations as the first component, suggesting potential for further exploration beyond Gaussian copulas.
  • Sparse Covariance Structures: The paper uses a factor structure for covariance matrices but suggests that extending the approach to other sparse covariance structures (e.g., sparse Cholesky factorization) is a natural next step.

Summary updated

BibTeX
@article{gunawan2024flexible,
  author = {David Gunawan and Robert Kohn and David J. Nott},
  title = {{Flexible Variational Bayes Based on a Copula of a Mixture}},
  journal = {J. Comput. Graph. Stat.},
  year = {2024},
  doi = {10.1080/10618600.2023.2262080},
}
Figure