TL;DR
This paper reframes Variational Inference (VI) as an optimization problem over distributions of variational parameters, introducing Wasserstein Gradient Descent (WGD) as a unified framework for VI. Their proposed methods, GFlowVI and NGFlowVI, offer more flexible and efficient ways to approximate complex posterior distributions, outperforming existing methods on synthetic and real-world datasets.
What Problem Does This Paper Address?
Variational Inference (VI) is a powerful technique for approximating intractable probability distributions, commonly encountered in Bayesian inference. Traditional VI methods, like Black-Box Variational Inference (BBVI) and Natural Gradient Variational Inference (NGVI), optimize the parameters of a chosen variational distribution. However, these methods can struggle with complex, multi-modal posterior distributions and may lack flexibility. This paper addresses these limitations by proposing a novel perspective on VI: instead of optimizing the variational distribution directly, they optimize a distribution over the variational parameters themselves. This allows for more expressive variational families, particularly mixtures of distributions, and provides a unified theoretical framework for existing VI methods.
Methods
The core contribution is the formulation of VI as an optimization problem in the space of probability distributions over variational parameters. They introduce Wasserstein Gradient Descent (WGD) as the optimization algorithm for this space. This approach allows them to:
- Unify BBVI and NGVI: They show that BBVI and NGVI can be seen as specific instances (particle approximations) of their proposed WGD framework.
- Propose GFlowVI and NGFlowVI: These are novel algorithms that implement the WGD approach. GFlowVI uses a standard gradient flow, while NGFlowVI incorporates a preconditioning matrix (related to the Fisher Information Matrix) for potentially faster convergence.
- Handle Mixture Models: By optimizing over distributions of parameters, their framework naturally extends to approximating posteriors with mixture models, which are better suited for complex, multi-modal distributions.
- Introduce Mirror Descent for Weights: To further enhance the expressiveness and efficiency, they incorporate an infinite-dimensional Mirror Descent (MD) update for the weights of the mixture components, leading to improved performance.
- Address Parameter Constraints: They propose a method using mirror maps to handle constraints on variational parameters, such as ensuring positive-definite covariance matrices.
Datasets
The paper evaluates their methods on:
- Synthetic Datasets:
- A 4-cluster Gaussian mixture distribution.
- A banana-shaped distribution.
- An X-shaped mixture of Gaussians.
- Real-world Datasets:
- 'Australia scale' (classification task).
- 'Boston' (regression task).
- 'Concrete' (regression task). These datasets are used to train Bayesian Neural Networks (BNNs).
Key Insights and Ideas
- Reframing VI: The most significant insight is viewing VI not as optimizing parameters of a single distribution, but as optimizing a distribution of those parameters. This shift in perspective unlocks new theoretical and algorithmic possibilities.
- WGD as a Unifying Framework: The paper elegantly demonstrates how established VI methods like BBVI and NGVI can be derived as special cases of their WGD approach, providing a more cohesive understanding of these techniques.
- Mixture Models for Complex Posteriors: By optimizing over distributions of parameters, their method naturally handles mixture models, which are crucial for approximating multi-modal and complex posterior distributions that single-component variational families struggle with.
- MD for Weight Adaptation: Incorporating Mirror Descent for updating mixture component weights significantly improves performance, suggesting that adaptive weighting is key for capturing complex posteriors.
- Theoretical Guarantees: The paper provides theoretical analyses of their proposed gradient flows, offering insights into convergence properties.
Key Results
- Synthetic Data: NGFlowVI consistently showed faster convergence and better KL divergence reduction compared to other methods, including WVI and NGVI, especially when using a sufficient number of mixture components (K=10). Visualizations confirmed that their methods produced variational distributions that closely matched the target distributions.
- Real-world Data (BNNs):
- GFlowVI and NGFlowVI with weight updates (using MD) generally outperformed their counterparts without weight updates, demonstrating the benefit of adaptive component weights.
- NGFlowVI and GFlowVI achieved competitive or superior prediction accuracy (lower negative log-likelihood/mean squared error) compared to WVI, NGVI, and SVGD on the tested datasets. NGFlowVI was particularly strong on 'Boston' and 'Concrete'.
- The methods showed improved performance as the number of mixture components (K) increased, with K=10 or more being generally beneficial.
- Computational Efficiency: The proposed methods are noted to be more particle-efficient than SVGD, especially for high-dimensional problems.
Limitations and Future Work
- Diagonal Gaussians: The current algorithmic development is primarily focused on diagonal Gaussian distributions. The authors acknowledge that this might not adequately capture posteriors with rotated Gaussian shapes.
- Parameter Constraints: While they propose a solution for parameter constraints (like positive-definite covariance), the current implementation focuses on diagonal covariance for simplicity.
- Future Work: The authors explicitly state their intention to address full covariance Gaussians in future work. They also plan to explore more complex constraints and potentially more advanced optimization techniques.
Summary updated