Variational inference with continuously-indexed normalizing flows

Anthony L. Caterini, Robert Cornish, Dino Sejdinovic, Arnaud Doucet

Abstract

Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks. However, CIFs do not possess a closed-form marginal density, making them difficult to use in standard Variational Inference (VI). This paper addresses this limitation by embedding CIFs into an Auxiliary Variational Inference (AVI) framework. The resulting models overcome the topological limitations of standard normalizing flows, yielding superior approximate posteriors for complex distributions and improving the performance of deep generative models like Variational Autoencoders (VAEs).

Introduction

In Variational Inference (VI), the goal is to approximate a complex, intractable posterior distribution with a simpler, parameterized family. Normalizing Flows (NFs) are a popular choice for this task, transforming a simple base distribution through a series of invertible functions. However, standard NFs are strictly bijective, which limits their expressiveness. Specifically, they struggle to model target distributions with complex topologies, such as disconnected modes, because they cannot "tear" the probability space.

Continuously-Indexed Flows (CIFs) relax this bijectivity constraint by augmenting the flow with continuous auxiliary "index" variables. While this significantly increases expressiveness, it renders the marginal density intractable. This paper proposes a method to successfully train CIFs as approximate posteriors by leveraging auxiliary variables within the Evidence Lower Bound (ELBO) optimization process.

Methods

To bypass the intractable marginal density of CIFs, the authors utilize Auxiliary Variational Inference (AVI). While the marginal density is intractable, the joint density over the latent variables and the auxiliary variables is tractable.

Graphical models illustrating the forward sampling of latent variable Z and the backward inference of auxiliary variables U.

The framework introduces an auxiliary inference distribution to optimize a lower bound on the standard ELBO. Key components include:

  • Reparameterization: The AVI objective is rewritten as an expectation over the base noise and auxiliary variables, making it differentiable via stochastic gradient descent.
  • Multi-Layer Structure: For deep CIFs, a structured "backward" inference network mirrors the conditional independence of the forward model, maintaining low variance in the ELBO estimator.
  • Amortization: The framework extends to amortized inference, allowing the base and auxiliary distributions to be conditioned dynamically on input data for generative modeling.

Key Insights

Theoretical analysis (Proposition 3.1) demonstrates that a CIF trained with the auxiliary ELBO will perform at least as well as its baseline normalizing flow counterpart. By using auxiliary variables to dynamically rescale or shift the base distribution, CIFs effectively bypass the topological bottlenecks that plague standard bijective flows. This allows them to map a single Gaussian base distribution into multiple disconnected modes without mathematical contradiction.

Experimental Results

Toy Mixture of Gaussians

On a synthetic 2D dataset featuring 9 disconnected Gaussian modes, CIF-augmented Neural Spline Flows (CIF-NSF) successfully captured all modes regardless of initial noise variance. In contrast, baseline NSFs failed when the initial noise was poorly scaled. On a more challenging 16-component task, CIFs achieved a significantly better marginal ELBO ($-0.116 \pm 0.021$) compared to baseline NSFs ($-0.562 \pm 0.008$).

Approximate posterior density learned by a CIF-augmented flow on a 9-mode Gaussian mixture, demonstrating the ability to overcome topological constraints.

Generative Modeling

When integrated into VAEs, CIF-augmented flows consistently outperformed baseline flows. On the MNIST dataset, a CIF-augmented Masked Autoregressive Flow (CIF-MAF) achieved a test log-likelihood of $-90.87$, surpassing both the baseline Small MAF ($-91.98$) and Importance-Weighted Autoencoder (IWAE) baselines ($-93.14$). Similar improvements were observed on the Fashion-MNIST dataset.

Conclusion and Future Work

This work demonstrates that CIFs can be effectively utilized for variational inference through an AVI framework, providing a powerful tool for modeling complex posterior topologies. While currently limited to continuous variables, future research may extend this to discrete latent variables or apply the expressiveness of CIFs to domains like data compression and Approximate Bayesian Computation (ABC).

Summary updated

Abstract

Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks. CIFs do not possess a closed-form marginal density, a...
BibTeX
@inproceedings{caterini2021variational,
  author = {Anthony L. Caterini and Robert Cornish and Dino Sejdinovic and Arnaud Doucet},
  title = {{Variational inference with continuously-indexed normalizing flows}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=D1VP3ytBxd},
}
Figure