Abstract
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference (FAVI) provides exact posterior marginals in mean-field approximations and allows for "free" marginalization over complex nuisance variables simply by ignoring them during training.
Introduction
Bayesian inference remains the gold standard for reasoning under uncertainty, yet its application is often hindered by the intractability of calculating the posterior distribution—the probability of latent factors given observations. Traditional Variational Inference (VI) addresses this by optimizing a surrogate distribution to minimize the reverse KL divergence. However, this approach faces two significant hurdles: it typically requires an analytical, differentiable likelihood function, and it struggles with marginalization. In many real-world scenarios, we only care about a subset of variables, but standard VI requires explicitly modeling all dependencies, including hundreds of nuisance variables, which is computationally prohibitive.
Methods: Forward Amortized Variational Inference (FAVI)
We propose FAVI, a method that utilizes the forward KL divergence within a joint-contrastive framework. By minimizing the loss $\mathcal{L}_{FA} = -\mathbb{E}_{p(x,z)}[\log q(z|x)]$, FAVI offers several key advantages:
- Likelihood-Free Inference: Since the expectation is taken over the generative model $p(x, z)$, the method only requires the ability to sample from the model. This makes it compatible with complex "black-box" simulators where density functions are unknown.
- No Reparameterization Required: FAVI does not require backpropagating through the sampling process, allowing for use with non-differentiable simulators.
- Exact Marginals: Theoretically, in a fully factorized (mean-field) setting, the FAVI loss is minimized by the exact posterior marginals. Unlike reverse KL VI, which often underestimates uncertainty and collapses onto modes, FAVI is "zero-avoiding" and captures the full spread of the posterior.
- "Free" Marginalization: If a model contains nuisance variables $\xi$ that are not of interest, an inference network $q(z|x)$ can be trained by sampling $(z, \xi, x)$ and simply ignoring $\xi$. The network naturally learns to marginalize over these variables without explicit modeling.
Experimental Results
MNIST Latent Recovery
We evaluated FAVI against standard Variational Autoencoders (VAEs) and Adversarially Learned Inference (ALI) using the MNIST dataset. FAVI achieved significantly lower latent reconstruction error compared to reverse KL methods ($p < 0.001$). Furthermore, as shown in the posterior distributions, FAVI more accurately captures the ground truth compared to the mode-seeking behavior of traditional VI.
Chaotic Forecasting in the Lorentz System
To test FAVI's performance in complex dynamical systems, we applied it to the Lorentz system, a chaotic model of atmospheric convection. A FAVI-based "Bayesian Variational Forecaster" outperformed the Extended Kalman Filter (EKF) by a factor of 1.94x in terms of predictive probability. Crucially, the FAVI forecaster successfully captured the system's "bifurcation"—the characteristic two-winged structure of the Lorentz attractor—which the EKF is fundamentally unable to represent.
Zero-Shot Meta-Classification
FAVI was also applied to meta-learning by training an RNN to solve arbitrary classification tasks. When tested on UCI datasets (Diabetes, Boston House Pricing, and Breast Cancer Wisconsin), the FAVI meta-classifier was competitive with specialized models like Random Forests and AdaBoost. Notably, the FAVI network achieved this "out of the box" without being specifically trained on these datasets, demonstrating the power of marginalizing over a vast space of possible models.
Discussion and Limitations
While FAVI demonstrates robust performance, it is dependent on the quality of the generative simulator; discrepancies between the simulator and reality will bias the inference. Additionally, we observed that the meta-classifier's performance scaled poorly with increasing training samples, a limitation likely tied to the RNN architecture rather than the FAVI method itself. Future work will explore more sophisticated architectures, such as Transformers, to enhance scaling and accuracy.
Summary updated