TL;DR
This paper introduces a novel framework for Variational Inference (VI) by reframing it as an optimization problem over the space of variational parameters, using Wasserstein gradient flows. This approach unifies existing methods like Black-Box VI (BBVI) and Natural-Gradient VI (NGVI) and extends them to handle complex, multi-modal posterior distributions, offering improved efficiency and flexibility.
What Problem Does This Paper Address?
Variational Inference (VI) is a powerful technique for approximating intractable probability distributions, commonly encountered in Bayesian inference. Traditional VI methods, such as BBVI and NGVI, optimize a set of variational parameters to make a chosen variational distribution closely match the true posterior. However, these methods often struggle with complex, multi-modal posteriors and can be computationally expensive. The authors address these limitations by proposing a new perspective: instead of directly optimizing the variational distribution's parameters, they optimize a distribution over these parameters using Wasserstein gradient flows. This allows for more flexible approximations and a unified view of existing VI techniques.
Methods
The core of the paper is the proposal of Wasserstein Gradient Descent (WGD) applied to a distribution over variational parameters. This is framed as a Wasserstein gradient flow on the space of probability distributions of these parameters.
Key methodological contributions include:
- Reframing VI: The problem of approximating a target distribution $\pi(z|D)$ with a variational distribution $q(z|\lambda)$ is reframed as optimizing a distribution $p(\lambda)$ over the variational parameters $\lambda$. The objective function becomes $L(p) = \mathbb{E}_{p(\lambda)} \mathbb{E}_{q(z|\lambda)} [f(z) + \log q(z|\lambda)]$.
- Preconditioned Wasserstein Gradient Flow: The authors introduce a preconditioned WGD, allowing for user-defined quadratic forms (preconditioning matrices $C$) to guide the flow for improved efficiency.
- Unified Perspective: BBVI and NGVI are shown to be special cases of this WGD framework, particularly when using a single particle or specific preconditioning matrices.
- Practical Algorithms (GFlowVI and NGFlowVI): They develop concrete algorithms for numerically solving these gradient flows. GFlowVI uses an identity preconditioning matrix ($C=I$), while NGFlowVI uses the inverse Fisher Information Matrix (FIM) as $C$.
- Weight Update with Mirror Descent: To handle mixture models effectively, they propose updating the weights of the mixture components using an infinite-dimensional Mirror Descent (MD) algorithm, which is shown to improve convergence.
- Handling Parameter Constraints: A method using mirror maps is introduced to ensure that variational parameters, particularly variances, remain positive during optimization, addressing potential numerical instability.
Datasets
The paper uses the following datasets for evaluation:
- Synthetic Datasets:
- A 4-cluster Gaussian mixture distribution.
- A banana-shaped distribution.
- An X-shaped mixture of Gaussians.
- Real-world Datasets (for Bayesian Neural Networks):
- 'Australia scale' (classification)
- 'Boston' (regression)
- 'Concrete' (regression)
Key Insights and Ideas
- Variational Parameters as a Distribution: The most significant insight is viewing the optimization not on the parameters of a single variational distribution, but on a distribution over those parameters. This opens up new avenues for modeling complex posteriors.
- Unification of VI Methods: The framework elegantly connects BBVI and NGVI to a more general WGD, providing a theoretical underpinning for their behavior.
- Gradient Flows for VI: Applying the theory of gradient flows to the parameter space offers a principled way to derive and analyze VI optimization algorithms.
- Mirror Descent for Weights: Using Mirror Descent for updating component weights in mixture models is a novel approach that improves the flexibility and expressiveness of the variational approximation.
- Constraint Handling: The mirror map technique for ensuring positive variances is a practical solution to a common numerical issue in VI.
Key Results
- Faster Convergence: NGFlowVI and GFlowVI demonstrated faster convergence in KL divergence on synthetic datasets compared to WVI and NGVI, especially when using a sufficient number of mixture components (K=10).
- Improved Approximation: Visualizations of samples from the learned variational distributions closely matched the target distributions for both synthetic datasets.
- Competitive Performance on BNNs: On real-world Bayesian Neural Network tasks, GFlowVI and NGFlowVI achieved competitive or superior results in terms of prediction loss (negative log-likelihood and mean square error) compared to WVI, NGVI, and SVGD.
- Effectiveness of Weight Updates: Experiments showed that incorporating the Mirror Descent update for component weights (GFlowVI vs. GFlowVI-w/o-MD) led to improved prediction accuracy.
- Scalability: The proposed methods are more particle-efficient than SVGD, especially in high-dimensional settings, due to their focus on updating mixture component parameters rather than kernel-based particle interactions.
Limitations and Future Work
- Diagonal Gaussian Assumption: The current algorithmic development is limited to diagonal Gaussian distributions for mixture components. The authors acknowledge that this may not adequately capture complex posteriors with rotated covariances.
- Full Covariance Gaussians: Addressing full covariance Gaussians for mixture components is identified as a key direction for future work.
- Parameter Constraints: While a solution for positive variances is proposed, generalizing this to handle other parameter constraints (e.g., bounds on covariance matrices) is a potential area for further research.
- Non-convexity: The objective function $L(p)$ is non-convex, meaning that the gradient flows may converge to local optima rather than the global optimum. This is a general challenge in VI that remains.
Summary updated