Normalizing flows are a powerful class of generative models praised for their ability to perform exact likelihood evaluation and efficient sampling. They achieve this by transforming a simple base distribution $p_Z(z)$ (like a standard Gaussian) into a complex data distribution $p_X(x)$ via an invertible, differentiable mapping $f$. Using the classical change-of-variables formula, the probability density of the data can be computed exactly:
$$p_X(x) = p_Z(f^{-1}(x)) \cdot \left| \det \left( \frac{\partial f^{-1}(x)}{\partial x} \right) \right|$$
However, this mathematical elegance comes with a strict architectural constraint: the mapping $f$ must be a bijection. This means the latent space $\mathbb{R}^d$ and the data space $\mathbb{R}^D$ must share the exact same dimensionality ($d = D$).
While mathematically convenient, this requirement directly conflicts with the Manifold Hypothesis, which posits that real-world high-dimensional data (such as images or 3D point clouds) typically concentrates around a lower-dimensional curved surface (a manifold) embedded within the high-dimensional ambient space. When a standard bijective flow is forced to model such data, it suffers from a severe structural mismatch. It is forced to allocate probability mass across the entire ambient space, leading to numerical instability, blurred samples, and poor density estimation.
To resolve this conflict, a new paradigm has emerged: Injective Normalizing Flows. By replacing bijective transformations with injective (one-to-one) mappings where $d < D$, these models compress the latent space and confine the generated probability distribution strictly to a learned lower-dimensional manifold.
The Mathematical Challenge of Rectangular Jacobians
When we transition from a bijective mapping to an injective mapping $f: \mathbb{R}^d \to \mathbb{R}^D$ (where $d < D$), the Jacobian matrix $J_f(z) = \frac{\partial f(z)}{\partial z}$ is no longer square. It is a rectangular matrix of size $D \times d$.
Because a rectangular matrix does not have a standard determinant, the ordinary change-of-variables formula breaks down. Instead, computing the density on the manifold requires the generalized change-of-variables formula, which introduces a volume-change factor:
$$p_X(x) = p_Z(z) \cdot \det \left( J_f(z)^T J_f(z) \right)^{-1/2}$$
The core challenge of injective flows lies in this term. Computing the matrix product $J_f(z)^T J_f(z)$ and its determinant is computationally expensive, scaling poorly with both the ambient dimension $D$ and the intrinsic dimension $d$. Without clever architectural constraints, calculating this volume-change gradient during training becomes intractable.
Architectural Strategies for Tractable Volume Change
Researchers have developed several distinct architectural strategies to bypass the expensive computation of rectangular Jacobian determinants, making injective flows practical and efficient.
1. Zero-Padding and Invertible Blocks
One of the most intuitive ways to build an injective flow is to embed a low-dimensional vector into a higher-dimensional space using a fixed operation, and then apply standard square normalizing flow layers.
In Flows for simultaneous manifold learning and density estimation, Johann Brehmer et al. introduce $\mathcal{M}$-flows. Their architecture constructs an injective mapping by taking a $d$-dimensional latent vector, padding it with $D - d$ zeros, and then passing the resulting $D$-dimensional vector through a sequence of standard, square invertible transformations. Because the transformations following the padding are bijective, the volume-change factor can be computed using standard, efficient normalizing flow techniques, while the zero-padding guarantees that the final samples are strictly confined to a lower-dimensional manifold.
2. Conformal Embeddings
Another elegant solution is to restrict the injective mapping to be conformal (angle-preserving).
In Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds, Brendan Leigh Ross et al. combine a standard normalizing flow with a trainable conformal embedding. Because conformal maps preserve angles between vectors, their rectangular Jacobian satisfies a unique property: $J_f(z)^T J_f(z) = c(z)^2 I$, where $c(z)$ is a scalar-valued scaling function and $I$ is the identity matrix. This mathematical property collapses the expensive determinant of a $d \times d$ matrix into a simple scalar function:
$$\det \left( J_f(z)^T J_f(z) \right)^{1/2} = c(z)^d$$
This allows for exact density estimation on learned manifolds without requiring any expensive matrix approximations.
3. Orthogonal Noise Padding and Rectangular Matrices
Instead of using fixed zero-padding, some approaches incorporate learned noise to smoothly transition between dimensions.
In A Change of Variables Method For Rectangular Matrix-Vector Products, Edmond Cunningham et al. introduce a scalable method for integrating rectangular matrices directly into normalizing flows. By incorporating learned orthogonal noise, their framework allows the flow to dynamically expand or contract dimensions while maintaining a tractable and efficient calculation for probability density, outperforming fixed-noise or simple zero-padding baselines.
4. Expansive Injective Layers
Rather than relying on a single embedding step, some architectures embed injective, dimension-expanding layers throughout the network.
In Trumpets: Injective Flows for Inference and Inverse Problems, Konik Kothari et al. propose TRUMPETs, a class of generative models that utilize expansive injective layers to progressively map a low-dimensional latent space to a high-dimensional data space. This design yields fast inverses and exact likelihood evaluations.
Similarly, Anthony L. Caterini et al. in Rectangular Flows for Manifold Learning design injective normalizing flows that utilize specialized rectangular transformations alongside a computationally tractable formulation for calculating the volume-change gradient, enabling simultaneous manifold learning and density estimation.
Alternative Perspectives: Implicit and Projected Manifolds
While explicitly designing injective layers is a direct way to respect data geometry, other researchers have approached the problem by relaxing the injectivity constraint or leveraging projections.
Implicit Manifolds via Noise Perturbation
Instead of forcing the architecture to be lower-dimensional, one can train a standard square normalizing flow in a way that coaxes it to implicitly discover the manifold.
In ManiFlow: Implicitly Representing Manifolds with Normalizing Flows, Janis Postels et al. demonstrate that standard flows trained on data perturbed with small amounts of noise can serve as implicit manifold representations. In this framework, the continuous probability distribution's regions of maximum likelihood naturally correspond to the underlying lower-dimensional structure. The authors then use this implicit representation to project noisy or out-of-distribution samples back onto clean, well-defined data surfaces.
Ambient Space Projection
For data that lives on known, explicit geometries (like spheres or tori), one can run a flow in an unconstrained ambient space and project it down.
In Learning Distributions on Manifolds with Free-Form Flows, Peter Sorrenson et al. develop Manifold Free-Form Flows (M-FFF). Instead of building specialized, manifold-specific neural network layers, they operate a standard feed-forward network in a higher-dimensional Euclidean embedding space and project the results onto the target Riemannian manifold using tangent-space gradient estimation. This provides a universal interface for complex geometries without requiring custom architectures for every distinct manifold shape.
Theoretical Foundations and Advantages
Why go through the mathematical trouble of designing injective flows? The answers lie in strong theoretical guarantees regarding approximation capability, generalization, and computational efficiency.
Universal Approximation
A primary theoretical concern for any new generative architecture is whether it can represent arbitrary distributions. In Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows, Michael Anthony Puthawala et al. provide the first rigorous proof that injective flows composed of simple, expansive neural network layers can universally approximate both the geometry of a manifold and the probability density on it, provided the embedding dimension is sufficiently large to avoid topological knots.
Bypassing the Curse of Dimensionality
Standard generative models often suffer from the curse of dimensionality, where the data required to learn a distribution grows exponentially with the ambient dimension $D$. Injective flows escape this trap.
In On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds, Biraj Dahal et al. prove that when data lies on a low-dimensional manifold, the error in learning the distribution depends strictly on the intrinsic dimension $d$ rather than the massive ambient dimension $D$. By modeling the generative process as a pushforward operation from a low-dimensional source, these models inherently bypass the curse of dimensionality.
Practical Benefits: Regularization and Memory Efficiency
Beyond pure theory, constraining a model to a low-dimensional latent space acts as a powerful inductive bias.
In Regularized Autoencoders via Relaxed Injective Probability Flow., Abhishek Kumar et al. relax the strict dimensionality constraints of standard flows to bridge the gap between flow-based models and autoencoders. They derive a principled training objective grounded in probability flows that functions as a Regularized Autoencoder (RAE), enabling data compression and high-level feature extraction.
This structural constraint is particularly valuable in ill-posed inverse problems, such as medical imaging or compressed sensing. In Conditional Injective Flows for Bayesian Imaging, AmirEhsan Khorashadizadeh et al. demonstrate that restricting generative models to injective mappings acts as a natural regularizer for imaging tasks. Because the model cannot generate samples outside the learned low-dimensional manifold, it excludes physically impossible solutions. Furthermore, eliminating the need to track full-dimensional bijections drastically reduces memory and computational requirements, allowing high-resolution Bayesian inference to run efficiently on standard hardware.
Key Foundational Work
Regularized Autoencoders via Relaxed Injective Probability Flow. (2020) — contributed 2 concept elements to this direction.
Trumpets: Injective Flows for Inference and Inverse Problems (2021) — contributed 2 concept elements to this direction.
A Change of Variables Method For Rectangular Matrix-Vector Products (2021) — contributed 2 concept elements to this direction.