Smoothing Discrete Data for Continuous Probability Flows

This direction is split into other directions: variational dequantization. Its last version stays readable and citable here.

Version 1 of 137 papers51 conceptsfrozen 2026-10-04

Normalizing flows and diffusion models are built on the elegant mathematics of continuous distributions. They rely on the change of variables formula, a calculus-based transformation that requires the data to be differentiable and continuous. However, much of the world’s data is stubbornly discrete: pixels in an image are integers from 0 to 255, molecules are composed of distinct atom types, and social science surveys use categorical labels.

If you try to train a continuous model directly on discrete values, the model discovers a "cheat code." It can assign infinite probability density to those specific discrete points, creating narrow spikes that lead to training instability and mathematically invalid likelihoods. To bridge this gap, researchers use a technique called dequantization.

The Intuition of Smearing

Imagine a probability distribution as a pile of sand. Discrete data is like placing that sand into a few extremely thin, tall columns. A continuous model, which prefers smooth hills and valleys, struggles to represent these infinitely thin columns.

Dequantization "smears" the sand. By adding a small amount of noise to each discrete value, we transform a single point into a small volume. For example, if we have a pixel value of 128, we might add noise from the range $0, 1)$, turning that single integer into a continuous value anywhere between 128.0 and 129.0. As [Ivan Kobyzev and Marcus A Brubaker explain in Normalizing Flows: An Introduction and Review of Current Methods, this ensures the model sees a continuous distribution, preventing density collapse.

From Uniform to Variational Noise

The simplest form of this technique is Uniform Dequantization. While effective at preventing infinite density, it creates a new problem: the "box" problem. By adding uniform noise, we force the generative model to learn a distribution with sharp, flat edges at the boundaries of each integer. Neural networks, which prefer smooth functions, find these sharp edges difficult to model.

In Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design, Jonathan Ho and colleagues introduced Variational Dequantization. Instead of adding simple uniform noise, they use a secondary, learned flow model to "shape" the noise. This allows the model to find the most natural way to smooth the discrete data into a continuous space, providing a tighter lower bound on the true discrete log-likelihood.

Handling Categorical Data without Order

Dequantization is straightforward when data has a natural order, like pixel intensities. But what about categorical data, like "Hydrogen" vs "Oxygen" in a molecule? Adding noise to arbitrary integer labels (1 for H, 2 for O) implies an order that doesn't exist.

To solve this, researchers like Phillip Lippe and Efstratios Gavves in Categorical Normalizing Flows via Continuous Transformations map categories into unique, non-overlapping volumes in a continuous latent space. Similarly, Emiel Hoogeboom and Max Welling developed Argmax Flows, which treat categorical labels as partitions of a Euclidean space. By learning a probabilistic "lifting" from discrete categories to these continuous regions, the model avoids imposing artificial ordering.

Aligning Noise with the Model Schedule

In the context of Diffusion ODEs, the choice of dequantization noise can be even more specialized. Kaiwen Zheng and Jun Zhu demonstrated in Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs that using a Truncated-Normal Dequantization can outperform learned variational methods.

The logic is one of alignment: if the generative model itself uses Gaussian noise during its diffusion process, the dequantization noise should match that characteristic. By sampling noise from a truncated normal distribution that fits the model's inherent noise schedule, researchers can eliminate the mismatch between how data is prepared and how the model perceives it, leading to state-of-the-art density estimation.

Why It Works

Dequantization works because it transforms a representation problem into an optimization problem. Instead of forcing a continuous model to handle the "illegal" geometry of discrete points, we provide a continuous "proxy" that the model can navigate using standard gradients. Whether through simple uniform noise in GraphNVP or complex surjective transformations in SurVAE Flows, the goal remains the same: smoothing the discrete world so that continuous probability flows can move through it.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_smoothing_discrete_data_for_continuous_p_1c75fdd5,
  title = {Smoothing Discrete Data for Continuous Probability Flows},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/smoothing-discrete-data-for-continuous-probability-flows/rdr_1c75fdd5f7214724ade0dd619c9bf2a4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_smoothing_discrete_data_for_continuous_p_1c75fdd5_v1,
  title = {Smoothing Discrete Data for Continuous Probability Flows},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/smoothing-discrete-data-for-continuous-probability-flows/rdr_1c75fdd5f7214724ade0dd619c9bf2a4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}