Generative diffusion models have achieved remarkable success in continuous domains like image and audio synthesis. These models work by gradually injecting Gaussian noise into data and learning to reverse this process. However, much of the world’s data is inherently discrete: text consists of categorical tokens, molecules are defined by specific bonds, and graphs are composed of "all-or-nothing" edges.
Applying continuous diffusion frameworks to these discrete structures creates a fundamental representation mismatch. Researchers are currently bridging this gap by redesigning the mathematical foundations of diffusion to natively handle categorical transitions.
The Problem with Blurry Categories
In a standard diffusion model, data is corrupted by adding small amounts of Gaussian noise. For a continuous pixel value, this makes sense—a color can become slightly lighter or darker. But for discrete data, like a word in a sentence or an atom in a molecule, there is no "slightly different" state.
As noted by Martinkus et al., applying continuous Gaussian noise to discrete structures forces the model to waste significant expressive power. Instead of learning the underlying connectivity patterns of a graph, the model spends its capacity trying to resolve "blurry" intermediate values back into binary edges. This mismatch often leads to chemically invalid molecules or topologically sparse graphs that require complex post-processing to become usable, as argued by Xu et al..
Strategy 1: Discrete State Markov Chains
To avoid the "blurriness" of Gaussian noise, one major research direction replaces Stochastic Differential Equations (SDEs) with Continuous Time Markov Chains (CTMCs). In this framework, the "noise" is not a displacement in Euclidean space, but a probability of jumping between discrete states.
Campbell et al. demonstrated that shifting to a continuous-time framework for discrete denoising allows for more flexible sampling and stronger theoretical guarantees. By modeling the noising process as a jump process, the model learns to predict the rate at which one category (like a specific amino acid) transitions into another. This approach has been extended to graphs by Xu et al., whose DISCO model operates in a discrete-state, continuous-time setting to preserve topological sparsity without needing heuristic discretization.
Strategy 2: The Latent Bridge and Analog Bits
Another approach involves mapping discrete data into a continuous space where standard diffusion tools can function, then carefully mapping it back.
Chen et al. proposed Bit Diffusion, which represents discrete tokens as "analog bits"—real numbers that the model treats as continuous during the diffusion process but are rounded back to binary values during inference. This allows practitioners to use the highly optimized architectures designed for images on categorical data like text.
However, moving to a latent space can introduce its own set of problems. Chang et al. point out that for molecules, a standard latent space might not respect physical constraints, leading to the generation of chemically impossible structures. To solve this, the authors of GLAD (GLAD: Improving Latent Graph Generative Modeling with Simple Quantization) use a quantization step combined with "diffusion bridges" to ensure the model stays within the bounds of valid discrete latent points.
Unifying the Theory
Recent work has sought to prove that these discrete and continuous approaches are not just similar, but mathematically linked. Winkler et al. established a formal bridge using the Ehrenfest process, showing that discrete Markov jump processes and continuous diffusion models are two sides of the same coin. This allows researchers to transfer learned representations from continuous models to discrete domains.
Furthermore, Chandra et al. unified discrete, Gaussian, and simplicial diffusion under the umbrella of the Wright-Fisher population genetics model. This suggests that a single "universal" diffusion model could potentially handle multiple data types—categorical, continuous, and constrained—within a single mathematical framework.
Why It Works: Intuition from Energy and Geometry
The success of these methods stems from a shift in how we view the "denoising" task. In continuous space, denoising is often seen as following a gradient toward high-density regions. In discrete space, Chen et al. revealed that masked diffusion models are actually performing energy minimization in the space of discrete optimal transport.
By linking the mask schedule to the geometry of probability space, these models don't just "fill in the blanks"—they find the most efficient path to transform a random distribution of tokens into a structured sequence. This geometric perspective explains why discrete models, when properly formulated, can achieve high-quality results with far fewer sampling steps than their continuous counterparts.