Introduction
Discrete diffusion models, such as MaskGIT, have emerged as efficient alternatives to continuous diffusion models (e.g., Stable Diffusion) by operating on quantized image tokens rather than raw pixel values. However, despite their speed and compatibility with large language model architectures, they have historically struggled to match the visual fidelity and diversity of their continuous counterparts.
ReDDiT (Rehashing Noise for Discrete Diffusion) addresses this performance gap. By introducing a randomized, multi-index "rehashing noise" strategy and a theoretically grounded sampling mechanism, ReDDiT achieves state-of-the-art results on ImageNet-1K, matching the quality of continuous diffusion transformers while maintaining the efficiency of discrete frameworks.
The Challenge of Discrete Diffusion
In continuous diffusion, Gaussian noise provides a rich, high-dimensional signal that guides the model through a smooth denoising process. In contrast, discrete models typically rely on a single [mask] token to represent hidden information. This "absorbing state" approach is often too simplistic, providing a coarse signal that limits the model's expressive power.
Furthermore, existing discrete models often employ rigid, handcrafted "re-masking" heuristics like Gumbel-max sampling. These methods are difficult to tune and can become numerically unstable when dealing with the large vocabularies necessary for high-fidelity image generation, often resulting in low diversity or lost details.
The ReDDiT Approach: Rehashing and Principled Sampling
ReDDiT reimagines the "noising" and "denoising" pipeline through two key innovations:
1. Rehashing Noise
Instead of collapsing all masked tokens into a single state, ReDDiT expands the "absorbing states" to a set of multiple randomized indices ($m=1024$ in optimized models). This "multi-index corruption" creates a "noise cloud" that enriches the training signal. By providing a more nuanced representation of uncertainty, the model learns to navigate the data distribution with much greater precision, mimicking the diversity of continuous Gaussian noise.
2. Rehash Sampler
To generate images, the authors derived a new sampler based on discrete diffusion theory. Unlike standard models that deterministically "unmask" tokens based on scores, the Rehash Sampler utilizes a low-discrepancy categorical sampling approach. This allows the model to "refresh" or refine tokens iteratively during the generation process, ensuring that the final output adheres more closely to the true probability distribution of the training data.
Additionally, ReDDiT incorporates Representation Alignment (RepA). By aligning the internal features of the Diffusion Transformer (DiT) with a pre-trained vision encoder (DINOv2), the model learns the underlying structural semantics of images significantly faster, reducing the overall training burden.
Results and Performance
Evaluated on the ImageNet-1K benchmark at 256×256 resolution, ReDDiT sets a new standard for discrete visual generation. It achieved a geometric Fréchet Inception Distance (gFID) of 1.61, a massive improvement over the MaskGIT baseline of 6.18.
Crucially, this performance places ReDDiT on par with leading continuous models like DiT-XL/2 (2.27 gFID). In terms of efficiency, ReDDiT generates high-quality images in just 32–64 steps, making it significantly faster than standard continuous diffusion models and more scalable than autoregressive alternatives.
Conclusion
ReDDiT demonstrates that the limitations of discrete diffusion are not inherent to the discrete state space but rather to the simplicity of traditional masking and sampling strategies. By bridging the gap with "rehashing noise," this work paves the way for a unified framework that can handle both visual and linguistic data with high performance and efficiency. Future work will likely focus on reducing the dependency on specific visual tokenizers and further optimizing the inference speed of the Rehash Sampler.
Summary updated