ReDDiT: Rehashing Noise for Discrete Visual Generation

Tianren Ma, Xiaosong Zhang, Boyu Yang, Junlan Feng, Qixiang Ye

Introduction

Discrete diffusion models, such as MaskGIT, have emerged as efficient alternatives to continuous diffusion models (e.g., Stable Diffusion) by operating on quantized image tokens rather than raw pixel values. However, despite their speed and compatibility with large language model architectures, they have historically struggled to match the visual fidelity and diversity of their continuous counterparts.

ReDDiT (Rehashing Noise for Discrete Diffusion) addresses this performance gap. By introducing a randomized, multi-index "rehashing noise" strategy and a theoretically grounded sampling mechanism, ReDDiT achieves state-of-the-art results on ImageNet-1K, matching the quality of continuous diffusion transformers while maintaining the efficiency of discrete frameworks.

The Challenge of Discrete Diffusion

In continuous diffusion, Gaussian noise provides a rich, high-dimensional signal that guides the model through a smooth denoising process. In contrast, discrete models typically rely on a single [mask] token to represent hidden information. This "absorbing state" approach is often too simplistic, providing a coarse signal that limits the model's expressive power.

The discrete masking process involves hiding image tokens behind a uniform mask, which can lead to a loss of nuanced signal during training.

Furthermore, existing discrete models often employ rigid, handcrafted "re-masking" heuristics like Gumbel-max sampling. These methods are difficult to tune and can become numerically unstable when dealing with the large vocabularies necessary for high-fidelity image generation, often resulting in low diversity or lost details.

The ReDDiT Approach: Rehashing and Principled Sampling

ReDDiT reimagines the "noising" and "denoising" pipeline through two key innovations:

1. Rehashing Noise

Instead of collapsing all masked tokens into a single state, ReDDiT expands the "absorbing states" to a set of multiple randomized indices ($m=1024$ in optimized models). This "multi-index corruption" creates a "noise cloud" that enriches the training signal. By providing a more nuanced representation of uncertainty, the model learns to navigate the data distribution with much greater precision, mimicking the diversity of continuous Gaussian noise.

2. Rehash Sampler

To generate images, the authors derived a new sampler based on discrete diffusion theory. Unlike standard models that deterministically "unmask" tokens based on scores, the Rehash Sampler utilizes a low-discrepancy categorical sampling approach. This allows the model to "refresh" or refine tokens iteratively during the generation process, ensuring that the final output adheres more closely to the true probability distribution of the training data.

Comparison between traditional MVTM (top) and the proposed ReDDiT framework (bottom). ReDDiT uses softmax probabilities and a rehash sampler to produce more diverse and detailed images compared to the score-based heuristics of previous methods.

Additionally, ReDDiT incorporates Representation Alignment (RepA). By aligning the internal features of the Diffusion Transformer (DiT) with a pre-trained vision encoder (DINOv2), the model learns the underlying structural semantics of images significantly faster, reducing the overall training burden.

Results and Performance

Evaluated on the ImageNet-1K benchmark at 256×256 resolution, ReDDiT sets a new standard for discrete visual generation. It achieved a geometric Fréchet Inception Distance (gFID) of 1.61, a massive improvement over the MaskGIT baseline of 6.18.

Crucially, this performance places ReDDiT on par with leading continuous models like DiT-XL/2 (2.27 gFID). In terms of efficiency, ReDDiT generates high-quality images in just 32–64 steps, making it significantly faster than standard continuous diffusion models and more scalable than autoregressive alternatives.

Conclusion

ReDDiT demonstrates that the limitations of discrete diffusion are not inherent to the discrete state space but rather to the simplicity of traditional masking and sampling strategies. By bridging the gap with "rehashing noise," this work paves the way for a unified framework that can handle both visual and linguistic data with high performance and efficiency. Future work will likely focus on reducing the dependency on specific visual tokenizers and further optimizing the inference speed of the Rehash Sampler.

Summary updated

BibTeX
@inproceedings{ma2026reddit,
  author = {Tianren Ma and Xiaosong Zhang and Boyu Yang and Junlan Feng and Qixiang Ye},
  title = {{ReDDiT: Rehashing Noise for Discrete Visual Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=7R8ohzWB4i},
}
Figure