Continuous generative frameworks like diffusion and flow matching have revolutionized image and video synthesis, but their reliance on Gaussian noise makes them naturally ill-suited for discrete data. To bridge this gap, researchers have converged on a technique known as analog bits: representing discrete categorical labels as bit strings and casting them into a real-valued space. This allows models to treat categorical data as continuous signals during training while preserving the underlying discrete structure.
Mapping Discrete Data to Continuous Space
The core challenge in applying diffusion to discrete tasks—such as panoptic segmentation, action anticipation, or layout generation—is that the standard denoising objective assumes a continuous manifold. Lala Li, Geoffrey E. Hinton, and their colleagues addressed this in A Generalist Framework for Panoptic Segmentation of Images and Videos by converting discrete labels into binary bits. These bits are then treated as "analog" values (e.g., -1 and 1) and subjected to standard continuous diffusion processes.
By treating bits as analog signals, the model can leverage the powerful denoising capabilities of frameworks like Pix2Seq-D. During inference, the model generates continuous values that are simply thresholded back into discrete bits to recover the original categorical labels.
Enhancing Stability and Speed
This continuous representation offers several advantages over traditional autoregressive or purely discrete generative models:
- Smoother Trajectories: In LayoutFlow: Flow Matching for Layout Generation, Julian Jorge Andrade Guerreiro and Hideki Nakayama demonstrate that analog bits allow flow matching to learn straight-line trajectories between noise and data. This results in faster generation and better adherence to design constraints compared to iterative diffusion.
- Uncertainty Modeling: For temporal tasks like action anticipation, Olga Zatsarynna and Juergen Gall show in Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation that mapping one-hot vectors to continuous space enables the model to jointly capture the uncertainty of both past observations and future predictions.
- Holistic Generation: In the domain of sketches and handwriting, Ayan Das and Timothy Hospedales use analog bits in ChiroDiff to model binary pen states. This non-causal approach allows the model to refine the "big picture" of a drawing globally, rather than predicting it stroke-by-stroke.
Why It Works
The intuition behind analog bits is that they provide a "soft" version of discrete choices. Instead of forcing a model to jump between rigid categories, the continuous space allows the model to be "undecided" during the early stages of generation. As the denoising process progresses, the model gradually pushes these analog values toward the corners of a hypercube, effectively "deciding" on a discrete bit string. This hybrid approach combines the mathematical elegance of continuous diffusion with the precision required for discrete data structures.