Most of our powerful machine learning tools—from Diffusion models to Variational Autoencoders (VAEs)—are built on the mathematics of continuity. They rely on smooth gradients, real-valued latent spaces, and probability density functions that assume data can exist anywhere on a line.
However, real-world data is often stubbornly discrete. We count infections, rank players, and store images as 8-bit integers. When we force these discrete "points" into continuous "spaces," we create a fundamental model-data mismatch. This mismatch isn't just a theoretical nuisance; it leads to infinite densities, nonsensical predictions (like negative counts), and poor compression.
The Density Trap and Dequantization
If you try to fit a continuous probability density function (PDF) to discrete data points, the model discovers a "cheat code": it can achieve infinite likelihood by placing a sharp spike (a Dirac delta) exactly on each observed integer. As Emiel Hoogeboom and colleagues point out in Learning Discrete Distributions by Dequantization, this prevents the model from learning the actual shape of the distribution, resulting in a degenerate solution that doesn't generalize.
To fix this, researchers use dequantization: adding a small amount of noise to discrete data to "smear" it into a continuous range. While uniform noise is common, Kaiwen Zheng and Jun Zhu show in Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs that the choice of noise matters. If the noise distribution doesn't align with the model's internal assumptions, you end up with a "likelihood gap" that degrades performance.
The Problem with Gaussian Counts
A common shortcut in regression is to treat counts (0, 1, 2...) as continuous numbers and use a Gaussian (Normal) distribution. This introduces a problematic inductive bias. As Spencer Young and Porter Jenkins argue in their work on Deep Double Poisson Networks, Gaussian models assign probability mass to impossible values—specifically negative numbers and non-integers.
This mismatch ruins calibration. If a model thinks there is a 10% chance of seeing -2.5 infections, its uncertainty estimates for the actual counts (0, 1, 2) are fundamentally skewed. To solve this, researchers are moving toward "fully heteroscedastic" discrete models that allow the variance to change based on the input without leaving the integer domain. Similarly, Christos Faloutsos and Flip Korn introduced the Discrete Gaussian Exponential (DGX) to fit long-tailed discrete data where traditional continuous approximations like the lognormal distribution fail to respect the integer-based nature of the data.
Bridging the Gap with Discrete Flows
For tasks like lossless compression, "close enough" isn't good enough. Continuous models like Normalizing Flows introduce quantization errors when mapping back to integers, making perfect reconstruction impossible.
Integer Discrete Flows (IDFs), developed by Emiel Hoogeboom and Max Welling, solve this by building bijections that map integers directly to integers. By staying within the discrete domain, these models avoid the rounding errors of continuous flows while maintaining the generative power of deep likelihood-based models.
Why Approximations Fail at the Limit
Even our most fundamental statistical laws struggle with the "jitter" of discreteness. The Central Limit Theorem (CLT) suggests that everything eventually looks like a smooth bell curve, but this ignores the discontinuous jumps in discrete data.
Dmitry Dolgopyat demonstrates in An error term in the Central Limit Theorem for sums of discrete random variables that higher-order statistical corrections (like Edgeworth expansions) eventually hit a "wall." Once the magnitude of the discrete jumps matches the size of the correction term, the approximation begins to oscillate wildly. This "jitter" is a physical limit: you cannot use a smooth polynomial to perfectly describe a staircase.
Fixing Pervasive Errors
Sometimes, the mismatch is simply a mathematical bug that has become standard practice. Gabriel Loaiza-Ganem and John P. Cunningham identified such an error in The continuous Bernoulli: fixing a pervasive error in variational autoencoders. Many VAEs use a discrete Bernoulli likelihood to model continuous pixel values (between 0 and 1). Because the Bernoulli distribution is only defined for 0 and 1, using it for values like 0.5 is mathematically unsound. By introducing the continuous Bernoulli, they provided a principled way to model $[0, 1]$-valued data, leading to sharper images and better-defined likelihoods.
By acknowledging the discrete nature of our data rather than smoothing it away, these researchers are building models that are not only more mathematically rigorous but also more accurate and reliable in the face of real-world "jitter."