Abstract
Normalizing flows have received a great deal of recent attention as they allow flexible generative modeling as well as easy likelihood computation. While a wide variety of flow models have been proposed, our theoretical understanding of what these models can—and cannot—represent has lagged behind. This paper provides a rigorous look at the "expressive power" of several fundamental flow classes, identifying both their universal capabilities in low dimensions and their significant limitations in high-dimensional spaces.
Introduction: The Invertibility Constraint
Normalizing flows (NFs) operate by transforming a simple base distribution (typically a Gaussian) through a series of invertible, differentiable mappings to match a complex target data distribution. The primary advantage of this approach is that the invertibility constraint allows for exact probability density calculation via the change-of-variables formula.
However, this same constraint is highly restrictive. Unlike standard neural networks, which can collapse or expand space arbitrarily to act as universal approximators, NFs must maintain a bijective mapping. This research addresses two fundamental questions regarding this constraint:
- Exact Transformation: Can a finite sequence of flow layers exactly transform one distribution into another?
- Approximation: If exact transformation is impossible, how many layers ($T$) are required to achieve a target Total Variation distance?
Methods: Topology Matching and Piecewise Construction
The authors focus on three specific "non-triangular" flow families: Planar flows, Sylvester flows, and Householder flows. To analyze these, they introduce a technique called Topology Matching. Rather than analyzing the functions directly, they examine the directional derivatives of the log-density. If a flow is too "simple," it cannot alter the local "shape" or topology of the density in certain directions, creating a fundamental mismatch between the source and the target.
For the approximation analysis, the authors utilize synthetic probability distributions, including a novel construction termed Tail-consistent Piecewise Gaussians. This construction is pivotal in proving universal approximation in one dimension.
Key Results
The 1D Success Story
In one dimension, planar flows are surprisingly powerful. Theorem 3.2 establishes that even with simple ReLU activations, a finite number of layers can approximate any density supported on a finite union of intervals. This demonstrates that the invertibility constraint is not a barrier to expressivity in low-dimensional settings.
The High-Dimensional Wall
As dimensionality ($d$) increases, the limitations of these flows become apparent. For ReLU-based Sylvester flows, "zero-gradient" conditions must be preserved; if a target distribution has a different number of peaks or local curvature than the source, a finite ReLU flow cannot bridge the gap.
Furthermore, many popular flows use "local" non-linearities like $tanh$ or $sigmoid$. These functions only significantly alter the distribution within a thin "slice" of the high-dimensional space. The authors prove that to move mass across a large $d$-dimensional volume, the required depth $T$ must grow polynomially with $d$. For specific target distributions, the required depth is $\Omega(d^4)$. In practical terms, a 100-dimensional dataset might require 100 million layers—a requirement that far exceeds current hardware capabilities.
Conclusion and Future Work
While planar and Sylvester flows are theoretically universal in 1D, they face a "high-dimensional wall" due to the locality of their non-linearities and topological constraints. This study focuses on non-triangular flows; however, most state-of-the-art models (like RealNVP or Glow) use triangular coupling layers. Future research should investigate whether specific combinations of tangent matrices or non-linearities can bypass these depth requirements and extend this "topology matching" framework to more complex architectures like iResNets or Neural ODEs.
Summary updated