Abstract
Flow-based models typically define a latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data space that they natively reside in, rather inhabiting a lower-dimensional manifold. In such scenarios, flow-based models are unable to represent data structures exactly as their densities will always have support off the data manifold, potentially resulting in degradation of model performance. To address this issue, we propose to learn a manifold prior for flow models that leverage the recently proposed spread divergence towards fixing the crucial problem; the KL divergence and maximum likelihood estimation are ill-defined for manifold learning. In addition to improving both sample quality and representation quality, an auxiliary benefit enabled by our approach is the ability to identify the intrinsic dimension of the manifold distribution.
Introduction: The Manifold Mismatch
Normalizing flows are a popular class of generative models that transform a simple probability distribution (like a Gaussian) into a complex one using a sequence of invertible functions. A fundamental assumption in standard flow architectures is that the latent space and the data space share the same dimensionality ($D$). However, real-world data—such as high-resolution images of faces or handwritten digits—rarely fills the entire high-dimensional ambient space. Instead, it typically concentrates on a lower-dimensional "manifold" ($K < D$).
When a standard flow is forced to model a thin manifold, two significant issues arise. First, the mathematical foundations of training—Maximum Likelihood Estimation (MLE) and KL divergence—become ill-defined because the data density is essentially zero everywhere except on the infinitely thin manifold. Second, the model often attempts to "curl up" the manifold to fill the $D$-dimensional latent prior. This "curling" wastes network capacity, distorts latent representations, and leads to poor sample quality.
Methodology: Spread Flows
To address the dimension mismatch, the authors introduce Spread Flows, which combine a learnable manifold prior with a novel training objective.
1. The Manifold Prior
Instead of the standard $D$-dimensional isotropic Gaussian prior, Spread Flows utilize a "generalized Gaussian" prior with a learnable covariance matrix $AA^T$. By allowing this matrix to be low-rank, the prior itself can collapse onto a $K$-dimensional subspace that matches the data's intrinsic geometry. This allows the flow to map the manifold directly to a lower-dimensional subspace in the latent domain, avoiding the "curling" effect.
2. Spread KL Divergence
To resolve the mathematical singularities associated with manifold learning, the authors employ spread divergence. This technique involves adding a small amount of Gaussian noise to both the empirical data distribution and the model's generated distribution. This "spreads" the thin manifolds into volumes that overlap in the ambient space, ensuring that the KL divergence is well-defined, differentiable, and stable for optimization. In this framework, noise acts as a fundamental bridge connecting singular manifold geometry to continuous probability densities.
3. Automatic Dimension Discovery
A unique advantage of this approach is its ability to identify the data's intrinsic dimensionality. Because the covariance matrix $A$ is learnable, the model "decides" how many dimensions are necessary to represent the signal. After training, the eigenvalues of $AA^T$ typically show a sharp "step" or gap; the number of significant eigenvalues corresponds to the intrinsic dimension $K$.
Experimental Evaluation
The researchers evaluated Spread Flows across a variety of synthetic and real-world datasets to validate both generative performance and diagnostic capabilities.
- Toy Manifolds: On 2D lines and 3D S-curves, the model successfully mapped the data to the appropriate lower-dimensional latent subspaces.
- Synthetic Benchmarks: In a 1024-dimensional "Fading Square" dataset where the only variation is intensity ($K=1$), the model correctly identified the single intrinsic dimension.
- Real-World Images: On the CelebA dataset (faces), Spread Flows achieved a Fréchet Inception Distance (FID) of 44.8, significantly outperforming the baseline Conformal Embedding Flows (CEF), which scored 105.1. For MNIST, the model estimated an intrinsic dimension between 11 and 14, aligning with previous mathematical estimates.
- Reconstruction Quality: The model demonstrated high-quality image reconstruction even after projecting data down to the learned intrinsic dimensions. This confirms that the dimensions discarded by the low-rank prior were indeed primarily noise, while the signal was preserved on the manifold.
Conclusion and Future Directions
Spread Flows offer a principled solution to the manifold modelling problem in generative flows. By leveraging spread divergence and learnable low-rank priors, the model improves sample quality and provides a tool for intrinsic dimension estimation.
Current limitations include the assumption of a single continuous manifold. Real-world datasets like MNIST, which contain distinct classes (digits 0-9), may be better represented as a collection of disconnected manifolds. Future work integrating "mixing priors" could allow Spread Flows to handle such multi-modal structures more effectively.
Summary updated