Introduction
Autoregressive (AR) models have long dominated the landscape of large language models, generating text sequentially from left to right. This structure makes applying Reinforcement Learning (RL) straightforward, as the probability of a sequence is simply the product of individual token probabilities. However, Diffusion Language Models (dLLMs) are emerging as a powerful alternative. By starting with a "cloud" of masks and gradually refining them into clear text, dLLMs offer unique flexibilities, such as parallel generation and the ability to fill in blanks.
Despite these advantages, dLLMs face a significant mathematical hurdle: calculating the exact likelihood of a full generated sequence is practically intractable. Previous attempts to apply RL to dLLMs relied on biased sequence-level approximations, leading to unstable training and poor reasoning performance. To bridge this gap and unlock "System 2" thinking capabilities for dLLMs, researchers have introduced Amortized Group Relative Policy Optimization (AGRPO).
The AGRPO Approach
AGRPO fundamentally shifts the RL optimization focus from the entire sequence to individual denoising steps. By treating the generation process as a multi-step journey rather than a single leap, AGRPO achieves theoretical alignment between training and inference while remaining computationally efficient.
The methodology relies on three core pillars:
- Multi-Step Markov Decision Process (MDP): Instead of treating a complete generated answer as a single action, AGRPO models each unmasking step as a distinct step in an MDP. This allows the use of exact, stepwise likelihoods that are mathematically tractable.
- Amortized Timestep Estimation: Reasoning traces in dLLMs can require hundreds of denoising steps. Because calculating gradients for every step is computationally prohibitive, AGRPO uses Monte Carlo sampling to select a small, representative subset of timesteps to update.
- Variance Reduction: To maximize the learning signal from the sampled steps, the algorithm employs low-discrepancy sampling to ensure coverage across the beginning, middle, and end of the generation process. Furthermore, it uses Entropy Importance Sampling to prioritize updates on "pivotal" steps where the model exhibits high uncertainty—often the exact moments where critical "thinking" occurs.
Key Results and Insights
The researchers evaluated AGRPO using the LLaDA-8B-Instruct base model across four challenging reasoning and planning benchmarks: GSM8K (grade school math), MATH (competition math), Countdown (arithmetic targeting), and Sudoku (planning under constraints).
The results demonstrate that AGRPO delivers dramatic improvements over both the base model and existing dLLM RL methods:
- Massive Accuracy Gains: The model achieved absolute accuracy jumps of +59.4% on Countdown and +69.7% on Sudoku.
- Parity with AR Models: On the GSM8K benchmark, the AGRPO-trained model reached 88.1% accuracy, achieving parity with specialized autoregressive reasoning models like DeepSeekMath-RL.
- Learnable Speed: Post-training with AGRPO makes the model significantly more robust. The trained model can generate accurate answers in 4x fewer steps than the base model. At the fastest inference settings (32 steps), AGRPO achieved an 8.1x improvement over the base LLaDA-8B model on GSM8K.
- Theoretical Alignment: The success of AGRPO highlights that aligning the training process with the step-by-step nature of inference is crucial for stable and effective dLLM optimization.
Limitations and Future Directions
While AGRPO marks a significant leap forward for diffusion-based reasoning, several challenges remain. First, dLLMs still lag behind the most advanced autoregressive models on highly complex tasks like the MATH dataset. Second, current dLLM architectures require a fixed sequence length to be set prior to generation, lacking the dynamic growth capabilities of AR models.
Additionally, dLLMs currently lack the efficient memory mechanisms (like KV caching) that make AR inference fast. Implementing KV caching for dLLMs is a critical next step to improve their practical deployment speeds. Finally, the authors note an exciting avenue for future work: exploiting the bidirectional nature of dLLMs. Because diffusion models can look "left and right" simultaneously, future iterations could be trained to take a problem and a final answer, and seamlessly fill in the missing reasoning steps in the middle.
Summary updated