Abstract: Flow-based generative models typically define a latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data-space that they natively reside in, rather inhabiting a lower-dimensional manifold. In such scenarios, flow-based models are unable to represent data structures exactly as their density will always have support off the data manifold, potentially resulting in degradation of model performance. In addition, the requirement for equal latent and data space dimensionality can unnecessarily increase complexity for contemporary flow models. Towards addressing these problems, we propose to learn a manifold prior that affords benefits to both sample generation and representation quality. An auxiliary benefit of our approach is the ability to identify the intrinsic dimension of the data distribution.
Introduction
Normalizing flows (NFs) are powerful generative models that map complex data distributions to simple latent spaces via invertible transformations. However, a fundamental design flaw exists in standard implementations: they assume the data natively fills the entire "ambient" space. In reality, high-dimensional data like images often resides on a lower-dimensional manifold.
When a standard NF tries to force this "thin" data to match a "thick" high-dimensional Gaussian prior, the latent space becomes contorted—a phenomenon known as "curling up." Mathematically, this makes Maximum Likelihood Estimation (MLE) and KL divergence ill-defined because the data doesn't have a proper density function in the higher-dimensional space. To address this, we introduce Spread Flows.
Methodology
Spread Flows bridge the gap between theory and manifold reality through three key innovations:
- Learnable Manifold Prior: Instead of a fixed standard Gaussian, we use a "generalized Gaussian" prior $N(0, AA^T)$, where $A$ is a learnable lower-triangular matrix. This allows the model to learn a low-rank covariance that effectively "flattens" the prior to match the data's manifold.
- Spread KL Divergence: To handle undefined KL divergence for singular distributions, we use "spread divergence." This involves convolving both the model and the data distributions with a small amount of Gaussian noise in the latent space, making their densities well-defined and computable.
- Volume-Preserving Flows: By using architectures where the Jacobian determinant is 1, we simplify the training objective. The flexibility of the learnable prior compensates for the restricted expressivity of the volume-preserving transformation.
Key Insights and Results
Automatic Dimension Discovery
One of the most elegant features of this approach is that the intrinsic dimension of the manifold ($K$) does not need to be specified beforehand. By examining the eigenvalues of the learned covariance matrix $AA^T$, we can identify the intrinsic dimension by counting how many eigenvalues are significantly larger than zero.
In experiments, the model successfully identified $K=1$ for a "Fading Square" dataset and $K=2$ for an S-curve. For real-world MNIST digits, it estimated the dimension to be between 11 and 14, aligning with previous mathematical estimates. For CelebA faces, it suggested an intrinsic dimension of approximately 2700.
Performance Benchmarks
Spread Flows demonstrate significant improvements in both representation and generation:
- Sample Quality: On the CelebA dataset, Spread Flows achieved a Frechet Inception Distance (FID) of 44.8, significantly outperforming the competing Conformal Embedding Flow (CEF) model, which scored 105.1.
- Reconstruction: Images can be accurately reconstructed even after projecting them down to the learned intrinsic dimension, proving that the model captures the essential structure of the data without the "curling" artifacts seen in standard flows.
Conclusion and Future Work
By allowing the prior to be low-dimensional, the model no longer wastes neural network capacity trying to stretch a thin manifold to fill a thick Gaussian. This leads to cleaner latent representations and sharper samples. While the current model assumes a single continuous manifold, future work could explore "mixture priors" to handle datasets with distinct, disconnected components (like the ten digits of MNIST). Additionally, while volume-preserving flows simplify the mathematics, exploring more general flow architectures under the spread divergence framework remains a promising avenue for increasing model expressivity.
Summary updated