Most high-dimensional data, from natural images to protein structures, does not actually fill the space it lives in. Instead, it concentrates around a low-dimensional "manifold" embedded within that high-dimensional ambient space. This is the Manifold Hypothesis.
Standard Normalizing Flows (NFs) struggle with this reality. Because they are built on bijective (one-to-one and onto) mappings, they require the latent space to have the exact same dimensionality as the data space. If you try to model a thin 2D surface in 3D space using a standard flow, the model is forced to assign some probability mass to the "empty" space off the manifold, or it becomes numerically unstable as it tries to compress a volume into a plane.
Injective Flows solve this by relaxing the bijection requirement. Instead of mapping $\mathbb{R}^D \to \mathbb{R}^D$, they learn an injective mapping from a low-dimensional latent space $\mathbb{R}^d$ to a high-dimensional ambient space $\mathbb{R}^D$ (where $d < D$).
The Geometry of Injection
The intuition behind an injective flow is simple: start with a low-dimensional coordinate system and "embed" it into a larger space.
In Flows for simultaneous manifold learning and density estimation, Johann Brehmer and Kyle Cranmer describe a concrete way to build these models, which they call $\mathcal{M}$-flows. They take a low-dimensional latent vector, pad it with zeros to match the ambient dimension, and then pass it through a series of standard invertible transformations. Because the transformations are invertible, the "zero-padded" structure is preserved as a clean, $d$-dimensional manifold that can twist and turn through the $D$-dimensional space without ever thickening into a volume.
This approach allows the model to do two things at once:
- Manifold Learning: It finds the "shape" of the data (the embedding).
- Density Estimation: It learns the probability distribution on that shape.
Bridging Flows and Autoencoders
For years, the generative modeling world was split. Autoencoders were great at dimensionality reduction but lacked a rigorous way to calculate exact likelihoods. Normalizing flows provided exact likelihoods but couldn't perform dimensionality reduction.
Abhishek Kumar, Ben Poole, and Kevin Murphy showed in Regularized Autoencoders via Relaxed Injective Probability Flow that injective flows are the bridge between these two worlds. By treating the encoder and decoder as parts of an injective flow, they derived a training objective that looks like a Regularized Autoencoder (RAE) but is mathematically grounded in the change-of-variables formula for probabilities. This allows for data compression and high-level feature extraction within a framework that still respects the laws of probability.
Theoretical Guarantees and Isometry
Can these simple injective layers actually represent any manifold? Michael Anthony Puthawala, Ivan Dokmanić, and their colleagues addressed this in Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows. They proved that injective flows are "universal approximators"—meaning they can represent any smooth manifold and any density on that manifold, provided the ambient space is large enough to avoid "knots" (topological self-intersections).
However, just representing a manifold isn't always enough; we often want the latent space to be "meaningful." If two points are close together in the real world, they should be close together in the latent space. Jianfu Zhang and Li Niu tackled this in Isometric Manifold Learning Using Hierarchical Flow. They introduced a hierarchical architecture that enforces isometry—ensuring that distances in the high-dimensional data space are preserved in the low-dimensional latent space. This makes the model significantly more effective for tasks like anomaly detection, where "distance" from the known data manifold is the primary signal.
Why It Works: The Change of Variables
The magic of any flow-based model is the change-of-variables formula, which relates the density of data $p(x)$ to the density of latents $p(z)$. In a standard bijective flow, this involves the determinant of the Jacobian matrix.
In the injective case, the Jacobian is no longer square (it's $D \times d$). To calculate the density on the manifold, these researchers use the Jacobian's singular values or the Gram matrix ($J^T J$). This allows the model to calculate the "volume" of the manifold as it stretches and shrinks through space. By maximizing this manifold-constrained likelihood, the model is forced to pull the manifold toward the data points and adjust the latent density to match the data's frequency.