Abstract: Normalizing flows have received a great deal of recent attention as they allow flexible generative modeling as well as easy likelihood computation. While a wide variety of flow models have been proposed, there is little formal understanding of the representation power of these models. In this work, we study some basic normalizing flows and rigorously establish bounds on their expressive power. Our results indicate that while these flows are highly expressive in one dimension, in higher dimensions their representation power may be limited, especially when the flows have moderate depth.
Introduction
Normalizing flows work by taking a simple probability distribution, such as a standard Gaussian, and applying a series of invertible, differentiable transformations to turn it into a complex data distribution. Because they are invertible, they allow for exact likelihood calculation—a significant advantage over other generative models like GANs. However, while their empirical success is well-documented, their theoretical representation power has remained a "black box." This paper addresses whether common flow architectures, such as Planar, Sylvester, and Householder flows, can actually transform a base Gaussian into any arbitrary target distribution.
Methodology
The authors analyze three fundamental classes of "base flows":
- Planar Flows: Simple transformations that shift and scale along a specific direction.
- Sylvester Flows: A higher-dimensional generalization of planar flows.
- Householder Flows: Linear transformations based on reflections.
Two main analytical strategies are employed. Topology Matching (Local Analysis) examines how the directional derivatives of the density change to determine if a flow can match the local "shape" of a complex target. Total Variation Bounds (Global Analysis) calculates the maximum "distance" a single layer can move a distribution toward a target, allowing for the estimation of the minimum depth (number of layers) required for a good approximation.
Expressive Power in 1D
A significant finding of this work is the "1D Superpower": in one dimension, even simple ReLU-based planar flows are universal approximators. Theorem 3.2 proves that these flows can approximate any density supported on a finite union of intervals in 1D. This is demonstrated using a novel construction of "tail-consistent piecewise Gaussians."
As shown in the figure above, the target density is decomposed into segments that the flow can map individually. This allows the model to turn a simple Gaussian into complex, multi-modal distributions with high precision.
High-Dimensional Bottlenecks
In dimensions $d > 1$, the representation power of these flows faces severe limitations. For Sylvester flows, there exists a "complementary subspace" where the transformation is essentially constrained. If a target distribution has complex features in that subspace, a shallow flow simply cannot create them.
For ReLU-based flows, the authors establish a "Topology Matching" rule: local directional derivatives of the log-density are preserved in a rigid way. This implies that a finite-depth ReLU flow cannot exactly transform one mixture of Gaussians (MoG) into another unless they already share specific geometric properties. Furthermore, for "local" non-linearities like $tanh$ or $sigmoid$, the required depth $T$ to approximate certain distributions grows polynomially with the dimension $d$. For high-dimensional data, this suggests that "moderate depth" flows are mathematically incapable of reaching the target.
Conclusion and Future Work
While simple normalizing flows are universal approximators in 1D, they face significant topological bottlenecks in higher dimensions that may require nearly infinite depth to overcome. Future research may investigate whether combining different types of flows (e.g., mixing Sylvester and Householder) or utilizing more complex "triangular flows" (like RealNVP or Glow) can bypass these bottlenecks more efficiently than simply increasing the depth of basic architectures.
Summary updated