Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs

Daehoon Gwak, Min-Sub Jung, Junwoo Park, Min‐Ho Park, ChaeHun Park, Junha Hyung, Jaegul Choo2025 · DOI

Abstract

Masked diffusion models (MDMs) offer a promising non-autoregressive alternative for large language modeling. Standard decoding methods for MDMs, such as confidence-based sampling, select tokens independently based on individual token confidences at each diffusion step. However, we observe that this independent token selection often results in generation orders resembling sequential autoregressive processes, limiting the advantages of non-autoregressive modeling. To mitigate this phenomenon, we propose Reward-Weighted Sampling (RWS), a novel decoding strategy that leverages an external reward model to provide a principled global signal during the iterative diffusion process. Specifically, at each diffusion step, RWS evaluates the quality of the entire intermediate sequence and scales token logits accordingly, guiding token selection by integrating global sequence-level coherence. This method selectively increases the confidence of tokens that initially have lower scores, thereby promoting a more non-autoregressive generation order. Furthermore, we provide theoretical justification showing that reward-weighted logit scaling induces beneficial rank reversals in token selection and consistently improves expected reward. Experiments demonstrate that RWS significantly promotes non-autoregressive generation orders, leading to improvements across multiple evaluation metrics. These results highlight the effectiveness of integrating global signals in enhancing both the non-autoregressive properties and overall performance of MDMs.

Introduction: The Sequential Trap in Diffusion Models

Most Large Language Models (LLMs) today are autoregressive (ARMs), predicting text one token at a time from left to right. While successful, this sequential approach can lead to cumulative errors where a single early mistake derails the entire sequence. Masked Diffusion Models (MDMs), such as LLaDA, offer a promising bidirectional alternative. They start with a sequence of [MASK] tokens and iteratively fill them in, theoretically allowing the model to use both forward and backward context.

However, standard decoding methods like confidence-based sampling often fall into a "position bias" trap. They tend to unmask tokens adjacent to already visible ones, resulting in a generation order that mimics a standard left-to-right process. This sequential behavior wastes the MDM's potential for global, non-linear reasoning.

Comparison of generation orders across different modeling paradigms.

Methods: Reward-Weighted Sampling (RWS)

To break this sequential bias, we introduce Reward-Weighted Sampling (RWS). RWS injects a "global conscience" into the decoding process by evaluating the quality of the entire potential sequence rather than just individual token confidences.

The Reward-Weighted Sampling (RWS) decoding pipeline.

The RWS process at each diffusion step follows these key stages:

  1. Potential Prediction: The model greedily predicts a full candidate sequence from the current masked state.
  2. Reward Evaluation: An external Reward Model (e.g., Skywork or Llama-RM) evaluates this candidate sequence and provides a global score.
  3. Logit Scaling: This reward score scales the raw prediction scores (logits) of the masked positions.
  4. Rank Reversal: Mathematically, this scaling induces "rank reversals." If the global reward is high, it boosts the confidence of tokens that initially had lower local scores, allowing the model to prioritize tokens that are globally important over those that are merely locally convenient.
  5. Guided Selection: The top $k$ tokens based on these reward-adjusted scores are unmasked, and the process repeats.

Key Insights: Breaking the Diagonal

We introduce the Generation Order Deviation (GOD) metric to measure how much the unmasking order deviates from a strict left-to-right sequence. A higher GOD score indicates more "non-autoregressive" behavior. While standard MDMs show a diagonal, sequential unmasking pattern, RWS creates a scattered, non-linear pattern, proving it successfully exploits bidirectional context.

Furthermore, RWS increases lexical diversity. By reducing local biases, the model explores more varied and creative generation paths without sacrificing coherence. This global guidance forces the model to care about the meaning of the whole sentence before committing to specific words.

Experimental Results

We evaluated RWS using LLaDA-8B-Instruct across several benchmarks:

  • RewardBench: RWS achieved an average win rate of ~59% across six different reward models compared to standard sampling.
  • HumanEval: RWS improved the Pass@1 score for code generation by up to +6.71% absolute over the baseline.
  • Keyword-Constrained Generation: In a new benchmark requiring models to incorporate specific "anchor keywords," RWS doubled the GOD score (from 0.95 to ~2.2) and reduced perplexity by 12-16%.
  • Multi-turn Dialogue: In evaluations using GPT-4 as a judge, RWS-generated responses were preferred in 60-65% of cases.

Conclusion and Future Work

Reward-Weighted Sampling effectively enhances the non-autoregressive properties of MDMs, leading to superior text quality and coherence. While RWS increases inference time by 21-33% due to the reward model calls, future work will explore "frequency ablation" (applying guidance every few steps) and adaptive scaling to optimize performance. These results highlight the critical role of global signals in unlocking the full potential of bidirectional language modeling.

Summary updated

Abstract

Masked diffusion models (MDMs) offer a promising non-autoregressive alternative for large language modeling.Standard decoding methods for MDMs, such as confidence-based sampling, select tokens independently based on individual token confidences at each diffusion step.However, we observe that this independent token selection often results in generation orders resembling sequential autoregressive processes, limiting the advantages of non-autoregressive modeling.To mitigate this pheonomenon, we propose Reward-Weighted Sampling (RWS), a novel decoding strategy that leverages an external reward model to provide a principled global signal during the iterative diffusion process.Specifically, at each diffusion step, RWS evaluates the quality of the entire intermediate sequence and scales token logits accordingly, guiding token selection by integrating global sequence-level coherence.This method selectively increases the confidence of tokens that initially have lower scores, thereby promoting a more non-autoregressive generation order.Furthermore, we provide theoretical justification showing that rewardweighted logit scaling induces beneficial rank reversals in token selection and consistently improves expected reward.Experiments demonstrate that RWS significantly promotes nonautoregressive generation orders, leading to improvements across multiple evaluation metrics.These results highlight the effectiveness of integrating global signals in enhancing both the non-autoregressive properties and overall performance of MDMs.
BibTeX
@misc{gwak2025reward,
  author = {Daehoon Gwak and Min-Sub Jung and Junwoo Park and Min‐Ho Park and ChaeHun Park and Junha Hyung and Jaegul Choo},
  title = {{Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1754},
}
Figure