Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, Alain Durmus

TL;DR

This paper reveals that standard Uniform Diffusion Models (UDMs) are implicitly optimized by a "leave-one-out" posterior rather than a standard denoiser, exposing a hidden mismatch in common training objectives. By deriving exact mathematical conversions between these representations and introducing a novel absorbing-state reformulation, the authors bridge the performance gap between uniform and masked diffusion models, delivering significant generation improvements without additional training costs.

What Problem Does This Paper Address?

Discrete diffusion models have become a powerful alternative to autoregressive models for generating text and structured data. They generally fall into two categories: Masked Diffusion Models (MDMs), which replace tokens with a special [MASK] symbol, and Uniform Diffusion Models (UDMs), which replace tokens with a random selection from the vocabulary. Recently, MDMs have empirically outperformed UDMs, leading researchers to favor them for large-scale language tasks.

The authors of this paper investigate why this performance gap exists. They discover a fundamental mathematical mismatch in how UDMs are parameterized. In MDMs, predicting the clean data given the noisy data aligns perfectly with the model's reverse dynamics. In UDMs, however, the standard "bridge plug-in" parameterization actually optimizes for a leave-one-out (LOO) posterior—a distribution that predicts a clean token at a specific position without using its own noisy observation, relying only on the surrounding context. This means standard UDM training objectives have been inadvertently fighting against their own parameterization choices.

Methods

To resolve this mismatch and revitalize uniform diffusion, the authors introduce several key innovations:

  1. Exact Conversion Formulas: The authors derive exact, invertible mathematical equations to convert between a standard denoiser, the leave-one-out posterior, and the score function. This allows practitioners to disentangle the training objective from the model's parameterization (e.g., training a leave-one-out predictor using standard cross-entropy).
  2. Informed Predictor-Corrector Sampling: Leveraging the LOO formulation, the authors design a training-free Gibbs-like corrector step. Because the model implicitly learns the LOO distribution during standard training, this corrector can be applied at inference time to refine samples without training an auxiliary model.
  3. Absorbing State Uniform Diffusion (AUDM & MUDM): The authors propose a new way to look at uniform diffusion by conditioning the process on a random sequence of absorbing states or latent transition times. This reformulation decomposes uniform diffusion into masked-diffusion-like operations, inheriting structural benefits like carry-over unmasking and a natural remasking mechanism while strictly preserving the UDM joint law.

Datasets

The methods were validated across multiple language modeling tasks and a structured discrete puzzle task:

  • OpenWebText (OWT): Used for primary training and generative evaluation (using the GPT-2 tokenizer with a sequence length of 1024).
  • One Billion Word (LM1B): Used to evaluate validation perplexity during training.
  • Zero-Shot Transfer Benchmarks: AG News, Lambada, PTB, PubMed, and WikiText.
  • Sudoku Completion: A $9 \times 9$ Sudoku puzzle dataset (50k split) used as a controlled, highly structured discrete baseline.

Key Insights and Ideas

  • The "Aha!" Parameterization Mismatch: The standard UDM training objective doesn't target a standard denoiser; it targets a leave-one-out distribution. This distinction is completely hidden in masked diffusion because the two parameterizations mathematically coincide on masked tokens, explaining why MDMs initially seemed much easier to optimize.
  • Marginals Aren't the Problem: The empirical gap between masked and uniform diffusion isn't caused by the choice of noise (masking vs. uniform corruption) itself. Instead, it is driven by parameterization and sampling design.
  • Free Inference Upgrades: Because the LOO target is structurally independent of the local noisy token, applying sampling heuristics like top-$p$ or temperature scaling to the converted LOO representation—rather than the raw denoiser—yields cleaner text generation at zero extra computational cost.

Key Results

  • Perplexity Improvements: On the OpenWebText dataset, switching to the LOO-Denoiser parameterization consistently improved validation perplexity throughout training. In zero-shot evaluations across datasets like Lambada and WikiText, the LOO variants consistently outperformed their standard denoiser counterparts.
  • AUDM Superiority: The newly introduced Absorbing Uniform Diffusion Model (AUDM) achieved the best likelihood results among all uniform diffusion variants. It outperformed standard UDM across all zero-shot datasets and even surpassed MDM on zero-shot transfers (e.g., achieving a perplexity of 61.37 on AG News compared to MDM's 64.57).
  • Sudoku Accuracy: On the structured Sudoku task, UDMs trained with the leave-one-out target achieved nearly perfect validation solve rates, significantly outperforming MDM baselines.

Limitations and Future Work

  • Empirical vs. Theoretical Gap: While the leave-one-out target consistently demonstrates superior generative frontiers empirically, a rigorous theoretical explanation for why it produces better sample quality remains an open question.
  • Absorbing-State Generation Frontiers: Although AUDM and its resampled variant (ReAUDM) significantly improve likelihood and match MDM performance, the absolute best generative frontiers in the paper were still achieved by pure LOO-parameterized UDMs.
  • Continuous Extensions: The authors suggest that future work could replace the one-hot absorbing tokens in AUDM with simplex-valued conditioning vectors to smoothly interpolate between different types of denoising posteriors within a single network architecture.

Summary updated

Abstract

Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not. We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective. We characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow us to disentangle parameterization and training objective. Our results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor. We further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism. On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design. The code and models can be found at https://github.com/samsongourevitch/rev_udm.
BibTeX
@inproceedings{gourevitch2026uniform,
  author = {Samson Gourevitch and Yazid Janati and Dario Shariatian and Umut Simsekli and Eric Moulines and Eric P. Xing and Alain Durmus},
  title = {{Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation}},
  booktitle = {SPIGM @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=BKpsDmhO1d},
}
Figure