Abstract
Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks. However, CIFs do not possess a closed-form marginal density, which prevents them from being directly used in variational inference (VI) schemes. We propose a method to use CIFs as expressive approximate posteriors by leveraging the framework of Auxiliary Variational Inference (AVI). By exploiting the conditional independence structure of multi-layer CIFs, we build auxiliary inference models that yield low-variance estimators of the model evidence. Our results demonstrate that CIFs significantly outperform standard flows in modeling distributions with complex topologies, such as multi-modal Gaussian mixtures and image datasets.
Introduction
Variational Inference (VI) is a cornerstone of modern Bayesian machine learning, relying on the expressiveness of the variational family to approximate complex posterior distributions. Normalizing Flows (NFs) have become a popular choice for this task due to their ability to transform simple distributions into complex ones via a series of invertible mappings. However, NFs are continuous bijections, which imposes strict topological constraints; for instance, they struggle to transform a single Gaussian mode into multiple disconnected components.
Continuously-Indexed Flows (CIFs) were introduced to overcome these limitations by incorporating auxiliary "indexing" variables. While this makes the model more flexible, it results in an "implicit" model without a closed-form marginal density $q(z)$, making the standard Evidence Lower Bound (ELBO) intractable.
Methodology: CIFs in Auxiliary Variational Inference
To integrate CIFs into the VI framework, we utilize Auxiliary Variational Inference (AVI). Instead of working with the intractable marginal $q(z)$, we define a joint distribution over the latent variable $z$ and an auxiliary variable $u$. The CIF structure naturally provides a way to define this joint distribution $q(z, u)$.
A key contribution of this work is the "natural synergy" between CIFs and AVI. We show that the required auxiliary inference model $r(u|z)$ can be constructed by reversing the layers of the CIF. This allows us to optimize a new lower bound that considers the joint space, effectively training the CIF to represent the target posterior.
Experimental Results
We evaluated the CIF-AVI framework on toy multi-modal distributions and high-dimensional image datasets.
Topological Flexibility
In a 9-mode Gaussian mixture experiment, we compared CIF-based Neural Spline Flows (CIF-NSF) against baseline NSFs. Standard flows are highly sensitive to the initial noise variance ($\sigma_0$). When $\sigma_0$ is low, standard flows often fail to capture all modes of the target distribution.
As shown in the figure, CIFs use the indexing variable to "clean up" the noise, successfully capturing all modes where the baseline fails. This confirms the theoretical advantage of CIFs in handling complex topologies.
Image Modeling
On MNIST and Fashion-MNIST, CIF-based models consistently outperformed their non-indexed counterparts. For example, on MNIST, CIF-MAF achieved a test-set log-likelihood of -82.70, surpassing both small and large standard MAF models. These gains were achieved with only a modest ~8.5% increase in parameter count, demonstrating the efficiency of the CIF approach compared to simply increasing the depth or width of standard flows.
Conclusion
This paper provides the necessary mathematical bridge to use CIFs in Variational Inference. By combining the topological flexibility of CIFs with the AVI framework, we create a powerful and parameter-efficient tool for approximating complex posteriors. Future work includes extending this framework to discrete data and exploring its utility in data compression and Approximate Bayesian Computation.
Summary updated