Dimensionality Flexible Normalizing Flows

This direction is merged into another direction: invertibility constraints, injective manifold mapping. Its last version stays readable and citable here.

Version 1 of 165 papers71 conceptsfrozen 2026-10-04

Normalizing Flows (NFs) are often described as the "holy grail" of generative modeling. Unlike Generative Adversarial Networks (GANs), which are notoriously difficult to evaluate, or Variational Autoencoders (VAEs), which rely on approximate lower bounds, NFs offer exact log-likelihood computation and efficient sampling. As noted in Normalizing Flows: An Introduction and Review of Current Methods, they achieve this by transforming a simple base distribution (like a Gaussian) through a sequence of invertible, differentiable mappings.

However, this mathematical elegance comes with a steep "bijectivity tax": the requirement that the latent space must have the exact same dimensionality as the input data. This constraint creates a fundamental tension with how we believe data actually works.

The Bijectivity Tax and the Manifold Hypothesis

In deep learning, we generally assume the Manifold Hypothesis: high-dimensional data (like a $256 \times 256$ image) actually resides on a much lower-dimensional manifold. Standard NFs, however, are forced to account for every single dimension. As Aditya Grover and Stefano Ermon demonstrate in Flow-GAN, this forces the model to waste representational capacity on noise and fine-grained details just to satisfy the requirement of an invertible, full-rank mapping.

This "dimension conservation" leads to several practical failures:

  1. Memory Bottlenecks: Because every layer must maintain the full dimensionality of the input, training deep flows on high-resolution data is incredibly memory-intensive, a problem addressed by tools like InvertibleNetworks.jl.
  2. Lack of Abstraction: Traditional autoencoders use a "bottleneck" to force the model to learn salient features. NFs cannot naturally discard irrelevant information, often resulting in latent representations that are cluttered with noise, as discussed in Neural PCA for Flow-Based Representation Learning.
  3. Manifold Mismatch: If data sits on a 2D plane inside a 3D space, a bijective flow will struggle to assign volume to that "thin" distribution.

Bridging the Gap with Surjections

To solve this, researchers have moved toward Surjective Flows. In SurVAE Flows, Didrik Nielsen and Max Welling introduced a framework that unifies VAEs and Flows. By using surjective transformations (many-to-one mappings), they allow the model to "shrink" dimensions.

The intuition is simple: instead of a strict bijection, the model uses a deterministic mapping for the important features and a stochastic process to handle the "discarded" information. This allows for operations like max-pooling or slicing, which are standard in other deep learning architectures but historically forbidden in flows. Carlo Albert and colleagues applied this in Surjective Sequential Neural Likelihood to automatically learn lower-dimensional representations of complex time-series data.

Injective Flows and Manifold Learning

While surjections help with dimensionality reduction, Injective Flows (one-to-many) help with manifold embedding. Abhishek Kumar and Kevin Murphy proposed Injective Probability Flow to bridge the gap between flows and regularized autoencoders. These models map a low-dimensional latent code into a high-dimensional space, effectively "inflating" the latent space to match the data's ambient dimension.

A clever alternative is SoftFlow, developed by Hyeongju Kim. Instead of changing the architecture, SoftFlow perturbs the data with a small amount of noise. This "thickens" the lower-dimensional manifold, giving it enough volume for a standard bijective flow to model it effectively. By conditioning the flow on the noise level, the model can still recover the clean, underlying structure.

Applications in Geometry and Physics

These flexible flows are particularly powerful for structured data like point clouds. In PU-Flow and its successor PU-INN, Aihua Mao and collaborators use invertible mappings to move between 3D coordinates and a latent space. By performing interpolation in the latent space and then "reversing" the flow, they can generate dense, uniform point clouds that maintain the original geometric topology.

Similarly, in the physical sciences, Neural Triangular Transport Maps by Youssef Marzouk use these principles to sample from complex lattice configurations, showing that the ability to map between spaces of different "effective" dimensions is crucial for scaling generative models to the constraints of real-world physics and geometry.

Papers in this version

Showing 60 of 65 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dimensionality_flexible_normalizing_flow_45e0b461,
  title = {Dimensionality Flexible Normalizing Flows},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimensionality-flexible-normalizing-flows/rdr_45e0b461e730430186408df645248af7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dimensionality_flexible_normalizing_flow_45e0b461_v1,
  title = {Dimensionality Flexible Normalizing Flows},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimensionality-flexible-normalizing-flows/rdr_45e0b461e730430186408df645248af7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}