Variational Inference with Continuously-Indexed Normalizing Flows

Anthony L. Caterini, Robert Cornish, Dino Sejdinovic, Arnaud Doucet

Abstract

Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks. However, CIFs do not possess a closed-form marginal density, which prevents them from being directly used in variational inference (VI) schemes. We propose a method to use CIFs as expressive approximate posteriors by leveraging the framework of Auxiliary Variational Inference (AVI). By exploiting the conditional independence structure of multi-layer CIFs, we build auxiliary inference models that yield low-variance estimators of the model evidence. Our results demonstrate that CIFs significantly outperform standard flows in modeling distributions with complex topologies, such as multi-modal Gaussian mixtures and image datasets.

Introduction

Variational Inference (VI) is a cornerstone of modern Bayesian machine learning, relying on the expressiveness of the variational family to approximate complex posterior distributions. Normalizing Flows (NFs) have become a popular choice for this task due to their ability to transform simple distributions into complex ones via a series of invertible mappings. However, NFs are continuous bijections, which imposes strict topological constraints; for instance, they struggle to transform a single Gaussian mode into multiple disconnected components.

Continuously-Indexed Flows (CIFs) were introduced to overcome these limitations by incorporating auxiliary "indexing" variables. While this makes the model more flexible, it results in an "implicit" model without a closed-form marginal density $q(z)$, making the standard Evidence Lower Bound (ELBO) intractable.

Methodology: CIFs in Auxiliary Variational Inference

To integrate CIFs into the VI framework, we utilize Auxiliary Variational Inference (AVI). Instead of working with the intractable marginal $q(z)$, we define a joint distribution over the latent variable $z$ and an auxiliary variable $u$. The CIF structure naturally provides a way to define this joint distribution $q(z, u)$.

Graphical models illustrating the sampling of latent variable $Z$ (left) and the auxiliary variables $U_{1:L}$ (right) within the proposed CIF-AVI framework.

A key contribution of this work is the "natural synergy" between CIFs and AVI. We show that the required auxiliary inference model $r(u|z)$ can be constructed by reversing the layers of the CIF. This allows us to optimize a new lower bound that considers the joint space, effectively training the CIF to represent the target posterior.

Experimental Results

We evaluated the CIF-AVI framework on toy multi-modal distributions and high-dimensional image datasets.

Topological Flexibility

In a 9-mode Gaussian mixture experiment, we compared CIF-based Neural Spline Flows (CIF-NSF) against baseline NSFs. Standard flows are highly sensitive to the initial noise variance ($\sigma_0$). When $\sigma_0$ is low, standard flows often fail to capture all modes of the target distribution.

Comparison of CIF-NSF (top) and baseline NSF (bottom) on a 9-mode Gaussian mixture with low initial noise ($\sigma_0 = 0.1$). The CIF-NSF achieves a significantly higher ELBO (-0.148 vs -0.625) and captures all modes.

As shown in the figure, CIFs use the indexing variable to "clean up" the noise, successfully capturing all modes where the baseline fails. This confirms the theoretical advantage of CIFs in handling complex topologies.

Image Modeling

On MNIST and Fashion-MNIST, CIF-based models consistently outperformed their non-indexed counterparts. For example, on MNIST, CIF-MAF achieved a test-set log-likelihood of -82.70, surpassing both small and large standard MAF models. These gains were achieved with only a modest ~8.5% increase in parameter count, demonstrating the efficiency of the CIF approach compared to simply increasing the depth or width of standard flows.

Conclusion

This paper provides the necessary mathematical bridge to use CIFs in Variational Inference. By combining the topological flexibility of CIFs with the AVI framework, we create a powerful and parameter-efficient tool for approximating complex posteriors. Future work includes extending this framework to discrete data and exploring its utility in data compression and Approximate Bayesian Computation.

Summary updated

Abstract

Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks. CIFs do not possess a closed-form marginal density, and so, unlike standard flows, cannot be plugged in directly to a variational inference (VI) scheme in order to produce a more expressive family of approximate posteriors. However, we show here how CIFs can be used as part of an auxiliary VI scheme to formulate and train expressive posterior approximations in a natural way. We exploit the conditional independence structure of multi-layer CIFs to build the required auxiliary inference models, which we show empirically yield low-variance estimators of the model evidence. We then demonstrate the advantages of CIFs over baseline flows in VI problems when the posterior distribution of interest possesses a complicated topology, obtaining improved results in both the Bayesian inference and surrogate maximum likelihood settings.
BibTeX
@inproceedings{caterini2021variational,
  author = {Anthony L. Caterini and Robert Cornish and Dino Sejdinovic and Arnaud Doucet},
  title = {{Variational Inference with Continuously-Indexed Normalizing Flows}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=D1VP3ytBxd},
}
Figure