Generative modeling often presents a difficult choice between exact density estimation and architectural flexibility. Normalizing flows offer a compelling solution by computing exact log-likelihoods and enabling fast, deterministic sampling. This elegance is made possible by a core mathematical mechanism: mapping a simple base distribution to a complex target distribution through a sequence of differentiable, bijective transformations, as introduced by Rezende et al..
To compute the exact probability density of a transformed variable, these models rely on the change-of-variables formula. This formula requires tracking how the probability space stretches or compresses, a factor quantified by the determinant of the transformation's Jacobian matrix. However, this mathematical requirement imposes severe architectural constraints, creating a fundamental tension between bijectivity and representational capacity, as discussed in the context of Unconstrained Monotonic Neural Networks.
The Bijective Dimensionality Constraint
In standard deep neural networks, layers routinely alter the dimensionality of intermediate representations. Computer vision architectures use pooling or strided convolutions to downsample spatial dimensions, while autoencoders compress high-dimensional inputs into a low-dimensional bottleneck to extract core features.
Because normalizing flows require strictly bijective mappings, they cannot alter dimensionality. A bijection maps elements one-to-one, meaning the latent space and all intermediate layers must maintain the exact same dimensionality as the input data, a constraint highlighted by Gritsenko et al. and Grcic et al..
This requirement introduces several practical limitations:
- No Dimensionality Reduction: The model cannot discard noise or compress data into a lower-dimensional embedding space, making it poorly suited for tasks like independent component analysis or compact representation learning, as noted by Li and Paige.
- Prohibition of Standard Layers: Common deep learning operations such as spatial downsampling, upsampling, or pooling are fundamentally non-invertible and cannot be directly integrated, as observed by Lee et al. and Wang et al..
- Computational and Memory Overhead: Operating entirely within the full-dimensional space of high-dimensional data (such as images) requires massive parameter counts and high memory consumption to capture complex dependencies, a challenge described by Grover et al..
The Jacobian Tractability Constraint
Beyond maintaining a strict dimensional match, invertible architectures must ensure that the log-determinant of the Jacobian matrix can be computed efficiently. For an arbitrary neural network layer mapping a $D$-dimensional input to a $D$-dimensional output, computing the determinant of its $D \times D$ Jacobian matrix scales cubically, $O(D^3)$. For high-dimensional data like images or audio, this computational cost makes training impossible.
To achieve a tractable Jacobian calculation—ideally scaling linearly, $O(D)$—the Jacobian matrix must be structured so that its determinant is trivial to compute. This is typically achieved by forcing the matrix to be diagonal or triangular, where the determinant is simply the product of the diagonal entries.
This mathematical constraint restricts the choice of neural network layers. As Schuurmans et al. and Si et al. explain, practitioners are often restricted to highly constrained functional forms, such as affine coupling layers. These structural limitations prevent the model from using standard, highly expressive non-linear layers, as noted by Djelouah et al..
For example, standard attention mechanisms or large-kernel convolutions cannot be easily incorporated because they lack simple Jacobian structures. Even when attention or global dependencies are integrated into flow architectures, they must rely on specific splitting and masking schemes to preserve tractability, which inherently limits the flexibility of the mechanism, as discussed in Generative Flows with Invertible Attentions.
Strategies for Bypassing Invertibility Constraints
To overcome the limitations imposed by bijectivity and tractability, recent research focuses on architectural modifications and alternative generative frameworks that relax these constraints.
Variational Padding and Dimension Changing
One approach to breaking the strict dimensionality bottleneck involves embedding the lower-dimensional data into a higher-dimensional space using noise. The authors of VFlow pad the input data with extra latent dimensions and optimize a variational lower bound, allowing the model to utilize high-dimensional intermediate representations. Similarly, Cunningham et al. incorporate learned orthogonal noise to allow rectangular (non-square) matrix transformations, enabling the flow to change dimensions while maintaining a tractable density calculation.
Shifting to Diffusion-Based Frameworks
Due to the rigid architectural compromises required by normalizing flows, a significant body of work has shifted toward diffusion-based models. Denoising Diffusion Probabilistic Models (DDPMs) and score-based models generate data through an iterative stochastic process rather than a single deterministic bijection.
By eliminating the requirement for exact architectural invertibility and tractable Jacobian determinants, diffusion frameworks allow the use of arbitrary, non-invertible neural network backbones, such as standard U-Nets or Transformers. This shift has been successfully applied to replace rigid flow-based priors in variational autoencoders (Wehenkel et al.), model complex time-series distributions (Rasul et al.), and construct highly expressive approximate posteriors for latent variable models (Piriyakulkij et al.).