Informed Correctors for Discrete Diffusion Models

Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, Scott Linderman

Discrete diffusion models have become a go-to framework for generating structured data like text, protein sequences, and tokenized images. However, they face a persistent "discretization" problem: during the sampling process, small approximation errors accumulate. Once a model makes a mistake in a discrete step—like picking the wrong letter in a word—it often has no way to "undo" that choice.

In their paper, "Informed Correctors for Discrete Diffusion Models," researchers introduce a way to let models fix their own mistakes by being smarter about how they "correct" their predictions. By pairing an informed sampling scheme with a novel Hollow Transformer architecture and a more efficient training objective, they achieve state-of-the-art results on text and image datasets with significantly fewer sampling steps.

The Problem: Accumulating Errors in Discrete Spaces

In continuous diffusion (like generating raw pixels), models can make tiny adjustments to "nudge" a sample toward the right distribution. In discrete diffusion, the model jumps between distinct categories or tokens. If a predictor step makes a mistake, standard "uninformed" correctors try to fix it by randomly re-masking tokens and trying again.

This is highly inefficient. It is akin to trying to fix a typo in a long essay by randomly deleting 10% of the characters and hoping you hit the mistake. The authors argue that the model already "knows" where it is likely to have messed up—it just needs a mechanism to act on that intuition.

Comparison of informed and uninformed correctors. The informed corrector targets the actual error (red box), while the uninformed corrector randomly masks and updates tokens (orange boxes).

The Solution: Informed Correctors and Hollow Transformers

The paper introduces three interconnected innovations to solve this inefficiency:

  1. Informed Selection: Instead of random re-masking, the "Informed Corrector" uses a confidence score (such as log-likelihood or the margin between the top two predictions) to rank tokens. It then applies an adaptive Gibbs sampling approach to update only the tokens the model is most "unsure" about.
  2. Hollow Transformer Architecture: To make the corrector efficient, the model needs to predict the value of a token at position $d$ without "seeing" the input at position $d$ (to avoid a trivial identity mapping). The authors designed a two-stream "Hollow Transformer" that achieves this "leave-one-out" property in a single forward pass.
  3. HollowDiff Training Objective: Standard discrete diffusion models typically only train to predict masked tokens. The authors realized the Evidence Lower Bound (ELBO) objective also provides a signal for unmasked tokens. They propose a combined loss that leverages both, acting as a variance reduction trick that speeds up training convergence.

Key Results

The authors evaluated their approach across multiple domains, including a synthetic Markov chain, character-level text (Text8), and tokenized ImageNet (256x256).

  • Higher Fidelity, Fewer Steps: On ImageNet 256x256, the "MD4 + Informed Corrector" achieved a state-of-the-art FID score of 5.78 using only 17 total steps. This outperformed the baseline MD4 (6.28) and the popular MaskGIT (6.56).
  • Cleaner Text: In the Text8 experiments, the informed corrector drastically reduced the word error rate. Qualitatively, the model produced far fewer "gibberish" words compared to standard discrete diffusion baselines.
Quantitative results on ImageNet 256x256 showing the MD4 + informed corrector achieving an FID of 5.78 in just 17 steps.

Limitations and Future Directions

While powerful, the approach has some constraints. The informed corrector relies heavily on the Hollow Transformer architecture, meaning it cannot be simply "plugged" into an existing, pre-trained standard Transformer without fine-tuning or re-training. Additionally, when updating many tokens at once, the model can sometimes "over-correct," leading to contradictions between tokens. Finding the optimal number of parallel updates remains a hyperparameter challenge.

Future work may involve scaling these "hollow" architectures to even larger language models or exploring more complex "locally-balanced" Markov processes for the corrector step, further bridging the gap between discrete and continuous diffusion capabilities.

Summary updated

Abstract

Discrete diffusion has emerged as a powerful framework for generative modeling in discrete domains, yet efficiently sampling from these models remains challenging. Existing sampling strategies often struggle to balance computation and sample quality when the number of sampling steps is reduced, even when the model has learned the data distribution well. To address these limitations, we propose a predictor-corrector sampling scheme where the corrector is informed by the diffusion model to more reliably counter the accumulating approximation errors. To further enhance the effectiveness of our informed corrector, we introduce complementary architectural modifications based on hollow transformers and a simple tailored training objective that leverages more training signal. We use a synthetic example to illustrate the failure modes of existing samplers and show how informed correctors alleviate these problems. On the Text8 dataset, the informed corrector improves sample quality by generating text with significantly fewer errors than the baselines. On tokenized ImageNet 256x256, this approach consistently produces superior samples with fewer steps, achieving improved FID scores for discrete diffusion models. These results underscore the potential of informed correctors for fast and high-fidelity generation using discrete diffusion.
BibTeX
@inproceedings{zhao2025informed,
  author = {Yixiu Zhao and Jiaxin Shi and Feng Chen and Shaul Druckmann and Lester Mackey and Scott Linderman},
  title = {{Informed Correctors for Discrete Diffusion Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=HDeIb67lJe},
}
Figure