The Expressive Power of a Class of Normalizing Flow Models

Zhifeng Kong, Kamalika Chaudhuri

Abstract

Normalizing flows have received a great deal of recent attention as they allow flexible generative modeling as well as easy likelihood computation. While a wide variety of flow models have been proposed, our theoretical understanding of what these models can—and cannot—represent has lagged behind. This paper provides a rigorous look at the "expressive power" of several fundamental flow classes, identifying both their universal capabilities in low dimensions and their significant limitations in high-dimensional spaces.

Introduction: The Invertibility Constraint

Normalizing flows (NFs) operate by transforming a simple base distribution (typically a Gaussian) through a series of invertible, differentiable mappings to match a complex target data distribution. The primary advantage of this approach is that the invertibility constraint allows for exact probability density calculation via the change-of-variables formula.

However, this same constraint is highly restrictive. Unlike standard neural networks, which can collapse or expand space arbitrarily to act as universal approximators, NFs must maintain a bijective mapping. This research addresses two fundamental questions regarding this constraint:

  1. Exact Transformation: Can a finite sequence of flow layers exactly transform one distribution into another?
  2. Approximation: If exact transformation is impossible, how many layers ($T$) are required to achieve a target Total Variation distance?

Methods: Topology Matching and Piecewise Construction

The authors focus on three specific "non-triangular" flow families: Planar flows, Sylvester flows, and Householder flows. To analyze these, they introduce a technique called Topology Matching. Rather than analyzing the functions directly, they examine the directional derivatives of the log-density. If a flow is too "simple," it cannot alter the local "shape" or topology of the density in certain directions, creating a fundamental mismatch between the source and the target.

For the approximation analysis, the authors utilize synthetic probability distributions, including a novel construction termed Tail-consistent Piecewise Gaussians. This construction is pivotal in proving universal approximation in one dimension.

Figure 5: Construction of a target distribution using piecewise components, a key step in proving 1D universal approximation.

Key Results

The 1D Success Story

In one dimension, planar flows are surprisingly powerful. Theorem 3.2 establishes that even with simple ReLU activations, a finite number of layers can approximate any density supported on a finite union of intervals. This demonstrates that the invertibility constraint is not a barrier to expressivity in low-dimensional settings.

Figure 1: Approximation of a target density $p$ by a planar flow $\tilde{p}$ in one dimension, illustrating the model's ability to capture multi-modal distributions.

The High-Dimensional Wall

As dimensionality ($d$) increases, the limitations of these flows become apparent. For ReLU-based Sylvester flows, "zero-gradient" conditions must be preserved; if a target distribution has a different number of peaks or local curvature than the source, a finite ReLU flow cannot bridge the gap.

Furthermore, many popular flows use "local" non-linearities like $tanh$ or $sigmoid$. These functions only significantly alter the distribution within a thin "slice" of the high-dimensional space. The authors prove that to move mass across a large $d$-dimensional volume, the required depth $T$ must grow polynomially with $d$. For specific target distributions, the required depth is $\Omega(d^4)$. In practical terms, a 100-dimensional dataset might require 100 million layers—a requirement that far exceeds current hardware capabilities.

Conclusion and Future Work

While planar and Sylvester flows are theoretically universal in 1D, they face a "high-dimensional wall" due to the locality of their non-linearities and topological constraints. This study focuses on non-triangular flows; however, most state-of-the-art models (like RealNVP or Glow) use triangular coupling layers. Future research should investigate whether specific combinations of tangent matrices or non-linearities can bypass these depth requirements and extend this "topology matching" framework to more complex architectures like iResNets or Neural ODEs.

Summary updated

Abstract

Normalizing flows have received a great deal of recent attention as they allow flexible generative modeling as well as easy likelihood computation. While a wide variety of flow models have been pro...
BibTeX
@inproceedings{kong2020expressive,
  author = {Zhifeng Kong and Kamalika Chaudhuri},
  title = {{The Expressive Power of a Class of Normalizing Flow Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=31chBQ898qK},
}
Figure