VFlow: More Expressive Generative Flows with Variational Data Augmentation

Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, Tian Tian

Introduction

Generative flows, such as Glow and Flow++, have emerged as powerful tools for density estimation and sampling due to their use of invertible transformations. However, these models face a fundamental architectural constraint: the "dimensionality bottleneck." Because invertibility requires every intermediate representation to maintain the same dimensionality as the input, flow models cannot easily utilize the "over-complete" features common in other deep learning architectures like ResNets or VAEs. Consequently, to achieve high expressiveness, standard flows must be extremely deep and computationally expensive.

VFlow addresses this limitation by introducing a variational data augmentation framework. By expanding the data space with additional latent dimensions, VFlow enables the use of wider internal representations, leading to more expressive and parameter-efficient generative models.

Methods: Breaking the Dimensionality Bottleneck

The core innovation of VFlow is the augmentation of the input data $\mathbf{x}$ with additional latent dimensions $\mathbf{z}$. This expands the data space from $D_X$ to $D_X + D_Z$, effectively widening the "pipe" through which information flows.

Architecture comparison: (a) standard flow vs. (b) VFlow with augmented dimensions $D_Z$.

The VFlow framework consists of several key components:

  1. Joint Modeling: A generative flow $p(\mathbf{x}, \mathbf{z}; \theta)$ is defined in the higher-dimensional augmented space.
  2. Variational Inference: Since $\mathbf{z}$ is latent, the model introduces a variational distribution $q(\mathbf{z}|\mathbf{x}; \phi)$ (an encoder) and optimizes the Evidence Lower Bound (ELBO).
  3. Theoretical Generalization: The authors prove that VFlow is a mathematical generalization of vanilla flows. Under mild assumptions, the maximum ELBO solution of a VFlow is guaranteed to be at least as good as the maximum likelihood solution of the corresponding vanilla flow.

This approach allows the model to "store" and "reuse" high-dimensional information across transformation steps, rather than being forced to re-extract features at every layer.

Key Results and Insights

VFlow was evaluated on toy datasets and standard image benchmarks, including CIFAR-10 and ImageNet.

Performance and Efficiency

VFlow achieved a state-of-the-art 2.98 bits per dimension (bpd) on CIFAR-10, surpassing previous leaders like Flow++ (3.08 bpd). Beyond raw performance, VFlow demonstrates remarkable parameter efficiency. For instance, a 6-channel VFlow achieved the same performance as a 3-channel Flow++ while using 2.6 times fewer parameters (11.9M vs. 31.4M).

Log-likelihood as a function of transformation steps and augmented dimensionality. Higher dimensions lead to better performance with fewer steps.

Geometric Interpretation

On toy datasets like the 2D Checkerboard, the authors observed that standard flows often struggle to map disjoint data modes into a compact latent space. VFlow overcomes this by using the extra dimensions to "lift" these modes. This allows the model to learn a "pile of pies" in the augmented space, where different modes are easily separated by the additional $\mathbf{z}$ dimensions, simplifying the overall transformation.

Conclusion and Future Work

VFlow successfully breaks the dimensionality bottleneck of generative flows, providing a path toward more efficient and expressive density models. While the current implementation uses a simple encoder for $q(\mathbf{z}|\mathbf{x})$, future work could explore more sophisticated variational distributions to further enhance performance. Additionally, the VFlow framework is general and could be applied to other flow architectures, such as Residual Flows or Continuous Normalizing Flows, to realize similar efficiency gains.

Summary updated

Abstract

Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints...
BibTeX
@inproceedings{chen2020vflow,
  author = {Jianfei Chen and Cheng Lu and Biqi Chenli and Jun Zhu and Tian Tian},
  title = {{VFlow: More Expressive Generative Flows with Variational Data Augmentation}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=Rh0TSSHgfg},
}
Figure