Standard diffusion models, like those powering Stable Diffusion, operate in a continuous world. They add small amounts of Gaussian noise to an image until it becomes a blur, then learn to reverse that blur. But much of the data we care about—segmentation masks, graph edges, or user-item interactions—is fundamentally discrete. When we force these binary "bits" into a continuous Gaussian framework, we introduce a mathematical mismatch: the model must learn to map continuous values back to discrete ones, often leading to instability and artifacts.
A growing body of research suggests a more natural alternative: Bernoulli Diffusion. Instead of blurring data with Gaussian noise, these models "corrupt" data by flipping bits.
The Forward Process: From Signal to Bit-Flips
In a standard diffusion model, the forward process $q(x_t | x_{t-1})$ adds Gaussian noise. In Bernoulli diffusion, the forward process is defined by a Bernoulli transition kernel. As Ze Wang and Ze Wang 0008 describe in Binary Latent Diffusion, this means that at each step, a bit has a certain probability of flipping its state (from 0 to 1 or vice versa).
As $t$ increases, the bits are flipped more frequently until the original structure is lost and the data reaches a stationary distribution—typically a uniform distribution where each bit has a 50% chance of being 0 or 1. This "bit-flip" noise is mathematically native to binary data. There is no need for thresholding or rounding because the data remains in its discrete state $\{0, 1\}$ throughout the entire process.
The Reverse Process: Denoising as Binary Classification
The magic of Bernoulli diffusion happens in the reverse process. Instead of predicting a "noise vector" to subtract, the model learns to predict the probability that a bit in its noisy state was originally a 0 or a 1.
Harshit Varma and colleagues demonstrate in the Glauber Generative Model that this simplifies the generative task into a series of binary classification problems: "Is this bit signal or noise?" By leveraging Glauber dynamics—a concept from statistical physics—they show that this approach can outperform complex variational methods.
Why It Works: Mathematical Alignment
The primary advantage of this approach is alignment. In medical imaging, for instance, Chenhui Wang and Hongming Shan argue in HiDiff that using a Bernoulli-based kernel prevents the degradation of binary structures. When refining a segmentation mask, a Gaussian model might produce "gray" intermediate values that don't correspond to any physical reality. A Bernoulli model, however, works directly with the discrete nature of the mask, leading to sharper boundaries and more stable training.
This alignment is equally critical in non-visual domains:
- Combinatorial Optimization: In DIFUSCO, Zhiqing Sun and Yiming Yang use Bernoulli noise to solve problems like the Traveling Salesman Problem. Flipping an edge from "included" to "excluded" aligns perfectly with the discrete decision-making required for graph-based tasks.
- Recommendation Systems: Gabriel Bénédict and the team behind RecFusion found that for sparse user-item interaction vectors, "directional" noise—flipping active bits to inactive—creates a smoother learning gradient for the model to reconstruct missing interactions.
Efficiency and Representation
Beyond accuracy, Bernoulli diffusion offers significant efficiency gains. Rong Xiao and Shaozhe Hao show in BiGR that by using compact binary latent codes instead of continuous ones, models can unify high-quality image generation with robust visual representation learning.
Because binary latent spaces are much smaller than continuous ones, these models can often sample much faster. For example, the Binary Latent Diffusion framework can generate high-resolution 1024x1024 images in as few as 16 steps, a massive leap over the hundreds of steps often required by traditional Gaussian diffusion models.
Selective Refinement
Finally, Bernoulli diffusion allows for "surgical" edits. In Uncertainty-Masked Bernoulli Diffusion, Yuqi Shen and Fengyang Xiao use the process to refine camouflaged object detection. By combining uncertainty quantification with Bernoulli noise, the model identifies only the "unsure" parts of a mask and applies bit-flips there, selectively repairing the segmentation without disturbing the parts the model is already confident about.