Executive Summary
Normalizing Flows are powerful generative models known for their ability to perform exact density estimation. However, they traditionally struggle when data lives on a low-dimensional manifold—such as a 2D surface embedded in a 3D space, or high-resolution images occupying a small subspace of their pixel dimensions. This paper introduces Conformal Embedding Flows (CEFs), a novel architecture that utilizes angle-preserving mappings to learn these manifolds while maintaining the ability to calculate exact, tractable probability densities. This achieves a computational feat previously thought to be prohibitive for injective models.
The Challenge: Manifold Hypothesis vs. Normalizing Flows
Traditional Normalizing Flows (NFs) are "bijective," meaning the latent space must possess the exact same number of dimensions as the data space. However, the "manifold hypothesis" suggests that high-dimensional data (like a $1024 \times 1024$ image) actually resides on a much lower-dimensional subspace.
Attempting to fit a standard NF to this type of data often causes the model to fail or become numerically unstable. The model is forced to spread probability mass over a high-dimensional volume that the data does not actually occupy. Previous attempts to remedy this—often referred to as "injective flows"—either sacrificed the ability to calculate exact densities or required extremely expensive computations (such as stochastic inverses or explicit Jacobian constructions) that fail to scale to large datasets like CelebA.
Conformal Embedding Flows (CEFs)
To bridge this gap, the authors propose Conformal Embedding Flows (CEFs). The core innovation lies in the use of conformal embeddings.
In geometry, a conformal map is one that preserves angles locally. Mathematically, this means the Jacobian matrix $J_g$ of the mapping satisfies: $$J_g^T J_g = \lambda^2(u) I$$ where $\lambda$ is a scalar factor and $I$ is the identity matrix.
Because of this specific geometric property, the complex determinant term in the change-of-variables formula—which is usually the computational bottleneck for density estimation in injective flows—simplifies into a simple scalar power of $\lambda$. This allows the model to map a low-dimensional latent base density onto a high-dimensional manifold while keeping the likelihood calculation efficient and exact.
Key Architectural Components
- Composable Conformal Layers: The authors provide a toolkit of layers including translations, rotations (orthogonal transformations), scalings, and a "Special Conformal Transformation" (SCT).
- Orthogonal $k \times k$ Convolutions: A new layer designed to scale these flows to image data by ensuring the convolution operation remains a conformal embedding.
- Joint Training: Unlike previous models that trained the manifold and the density separately, CEFs allow for a mixed loss function: $\text{Log-Likelihood} + \alpha(\text{Reconstruction Error})$.
Experimental Evaluation
The authors evaluated CEFs on a mix of synthetic and real-world benchmarks to test both their geometric flexibility and scalability:
- Spherical Data: A synthetic 2D distribution wrapped onto a 3D sphere.
- Synthetic CIFAR-10 Ship Manifolds: Images generated by a StyleGAN2-ADA model where the true manifold dimension is known (64 and 512 dimensions).
- MNIST: The classic handwritten digit dataset.
- CelebA: High-resolution celebrity face images, used to test the model's scalability to complex, high-dimensional manifolds.
Core Insights and Results
- Conformality is the "Sweet Spot": The authors argue that conformality is the minimal restriction needed for tractable manifold learning. Any looser condition makes density estimation intractable, while any tighter condition (like pure isometry) makes the model too rigid to learn complex shapes.
- Exact Likelihoods at Scale: CEFs are the first injective flow models to provide exact log-likelihoods on datasets as large as CelebA without requiring $O(n^3)$ computations.
- The Importance of Joint Optimization: By training the manifold (reconstruction) and the density (likelihood) simultaneously, the manifold can "deform" to make the density easier for the flow to learn. This leads to significantly better sample quality on complex datasets like CelebA.
- Scaling to Images: By introducing orthogonal convolutions, the authors prove that manifold-learning flows aren't just for toy 3D problems; they can handle the architectural requirements of modern computer vision.
Limitations and Future Directions
While CEFs represent a significant step forward, they come with inherent trade-offs:
- Expressivity Trade-off: Conformal mappings are a mathematically restricted class (governed by Liouville’s theorem). Consequently, CEFs are inherently less "flexible" than standard neural networks or non-conformal injective flows.
- Topology Constraints: The current formulation assumes the manifold is "diffeomorphic to Euclidean space." If the data lives on a complex shape with holes (like a torus), the model might encounter numerical instabilities.
The authors suggest that discovering even more expressive classes of parameterizable conformal embeddings is the next logical step to closing the performance gap with GANs and VAEs while retaining exact likelihoods.
Summary updated