Normalizing flows are exceptionally powerful generative models for continuous data. By passing a simple base distribution (like a standard Gaussian) through a series of invertible, differentiable transformations, they can warp and stretch a smooth probability density to fit highly complex data distributions.
Crucially, normalizing flows allow for exact likelihood evaluation. This tractability relies on the continuous change-of-variables formula:
$$p_X(x) = p_Z(f(x)) \cdot \left| \det J_f(x) \right|$$
Here, the Jacobian determinant $\left| \det J_f(x) \right|$ acts as a scaling factor, tracking how much the transformation $f$ stretches or compresses continuous volume at any given point.
However, this mathematical foundation introduces a fundamental conflict when applied to the real world: much of our data is discrete. Text tokens, pixel intensities, DNA sequences, graph structures, and low-volume counts do not live in a smooth, continuous space. They have no continuous volume to stretch, no derivatives to compute, and no Jacobians to evaluate. As pointed out by Keyon Vafa et al. in Discrete Flows: Invertible Generative Models of Discrete Data, the lack of continuous volume makes standard continuous flow mechanisms fundamentally incompatible with discrete data spaces.
To use normalizing flows on discrete data, researchers must bridge this continuous-discrete gap.
The Limits of Dequantization
The traditional workaround for modeling discrete data with continuous flows is dequantization. If your data consists of discrete integers (like pixel values from 0 to 255), you can add continuous noise—typically uniform noise $u \in 0, 1)$—to turn the discrete points into a continuous space.
While dequantization allows continuous flows to run without throwing mathematical errors, it introduces significant limitations:
- The Dequantization Gap: Adding noise forces the model to allocate probability mass to the spaces between integers. As discussed by [Alexey A. Gritsenko et al. in On the relationship between Normalising Flows and Variational- and Denoising Autoencoders, this requires specialized, complex dequantization schemes rather than a direct, unified approach. For tasks like lossless compression (Rianne van den Berg et al. in IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression) or low-volume count forecasting (Kashif Rasul et al. in Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows), this artificial smoothing complicates exact likelihood calculation and degrades performance.
- The Categorical Problem: Dequantization assumes the data is ordinal (i.e., that 1 is close to 2, and 2 is close to 3). But for categorical data—such as text tokens or image segmentation maps—there is no inherent ordering. Treating categories as numbers and adding noise implies an arbitrary spatial relationship that does not exist. As emphasized by Emiel Hoogeboom et al. in Argmax Flows: Learning Categorical Distributions with Normalizing Flows, standard dequantization methods introduce unwanted biases because the concept of "closeness" between categories cannot be captured by simple continuous mappings.
Native Discrete Transformations
To avoid the pitfalls of dequantization, one major branch of research focuses on building normalizing flows that operate natively on discrete spaces. Instead of scaling and shifting real numbers, these models use discrete algebraic operations that preserve probability mass exactly, bypassing the need for Jacobians entirely.
Continuous Flow: [Continuous Base] ----(Differentiable Warp)----> [Continuous Target]
Discrete Flow: [Discrete Base] ----(Discrete Permutation)---> [Discrete Target]
For integer-valued data, Keyon Vafa et al. demonstrated that invertible transformations can be constructed using modulo arithmetic. Similarly, Jakub M. Tomczak in General Invertible Transformations for Flow-based Generative Modeling proposed a general framework for constructing discrete coupling layers that handle integer values directly, improving performance on symbolic and integer datasets.
When data structures are more complex than simple integers, transformations must match the topology of the data:
- DNA and Tabular Sequences: Mai Elkady et al. introduced Discrete Tree Flows via Tree-Structured Permutations, which uses tree structures to define exact permutations over categorical domains.
- Categorical Labels: Alexandra Lindt et al. developed Discrete Denoising Flows, utilizing a denoising permutation mechanism to train discrete flows efficiently without relying on biased gradient estimators.
Bridging Spaces via Continuous Latents
An alternative strategy keeps the continuous flow intact but changes how the discrete data is mapped into the continuous space. Instead of adding heuristic uniform noise, these methods use learnable, structured interfaces.
Argmax and Variational Embeddings
Emiel Hoogeboom et al. introduced Argmax Flows, which pass a continuous flow through an argmax function to output discrete categories. To train this architecture, they define a "probabilistic right-inverse" that acts as a structured dequantizer, mapping categories back into continuous regions cleanly.
Expanding on this idea, Phillip Lippe et al. developed Categorical Normalizing Flows via Continuous Transformations. They use variational inference to learn a smooth embedding space for discrete categories, enabling parallel, permutation-invariant generation of complex structures like molecules.
Learnable Tessellations and Finite Volumes
Instead of mapping points to arbitrary coordinates, why not partition the continuous space to match the discrete data?
Ricky T. Q. Chen et al. proposed Semi-Discrete Normalizing Flows through Differentiable Tessellation, which uses learnable Voronoi diagrams to slice continuous space into convex polytopes. Each polytope corresponds to a discrete category, allowing the flow to learn the geometry of the discrete space dynamically.
Standard Dequantization: Learnable Tessellation (Voronoi):
+-------+-------+-------+ +---------+---+-----------+
| | | | | . | . | . |
| 1 | 2 | 3 | | +---+-----------+
| | | | | cat 1 |c2 | cat 3 |
+-------+-------+-------+ +---------+---+-----------+
(Rigid, uniform grids) (Flexible, learnable boundaries)
Similarly, Didrik Nielsen et al. introduced subset flows in Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow. By designing flows that can tractably compute the transformation of finite volumes rather than infinitesimal points, the authors closed the dequantization gap and allowed autoregressive models to be framed as exact, single-layer flows.
Open Frontiers: Symmetries and Causality
The continuous-discrete bottleneck is not just a theoretical problem; it limits the deployment of normalizing flows in highly structured downstream domains.
Physical Symmetries
In molecular modeling, nodes represent discrete atom types while coordinates represent continuous 3D positions. As highlighted by Victor Garcia Satorras et al. in E(n) Equivariant Normalizing Flows, mapping these discrete categories into a continuous space while strictly preserving physical symmetries (like rotation and translation equivariance) requires specialized smoothing techniques to ensure the flow remains invertible and stable.
Causal Inference
Normalizing flows are widely used to learn Structural Causal Models (SCMs) because their invertibility allows for exact counterfactual reasoning. However, real-world causal datasets are frequently mixed, containing both continuous measurements and discrete or categorical variables.
The authors of Causal normalizing flows: from theory to practice (Adrián Javaloy et al.) and DeCaFlow: A deconfounding causal generative model (Alejandro Almodóvar et al.) point out that current causal flow frameworks remain heavily optimized for continuous variables. When forced to handle discrete variables, they rely on approximations that can compromise causal validity.
As noted by Minh Khoa Le et al. in Learning Structural Causal Models from Ordering: Identifiable Flow Models, extending identifiable flow architectures to natively process discrete and categorical variables remains an active and critical frontier for the field.
Key Foundational Work
Argmax Flows: Learning Categorical Distributions with Normalizing Flows (2020) — contributed 3 concept elements to this direction.
On the relationship between Normalising Flows and Variational- and Denoising Autoencoders (2019) — contributed 1 concept elements to this direction.
Discrete Flows: Invertible Generative Models of Discrete Data (2019) — contributed 1 concept elements to this direction.