QuRL: Efficient Reinforcement Learning with Quantized Rollout

Yuhang Li, Reena Elangovan, Xin Dong, Priyadarshini Panda, Brucek Khailany

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.

Introduction: The Rollout Bottleneck

Training "reasoning" models, such as OpenAI’s o1 or DeepSeek-R1, relies heavily on Reinforcement Learning with Verifiable Rewards (RLVR). In this paradigm, models generate long Chain-of-Thought (CoT) sequences that are subsequently verified for correctness. However, the autoregressive nature of LLM decoding makes the rollout phase—where the model generates these responses—the primary efficiency bottleneck, often consuming up to 70% of the total training time.

While quantization (e.g., INT8 or FP8) is a standard technique for accelerating inference, applying it within the RL training loop presents significant challenges:

  1. Training Collapse: Divergence between a quantized "actor" (generating text) and a full-precision "learner" (updating weights) leads to mathematical instability in algorithms like PPO.
  2. The Weight Update Problem: RL updates are frequently as small as $10^{-7}$, whereas quantization errors are relatively large. Standard quantization methods often "mask" these tiny updates, effectively freezing the model and halting the learning process.

QuRL: Quantized Reinforcement Learning

To address these bottlenecks, we propose QuRL, a framework that utilizes a quantized actor for the rollout phase while maintaining full-precision parameters for gradient updates.

The QuRL framework decouples the quantized rollout from the full-precision weight updates to accelerate training.

Core Innovations

  • Adaptive Clipping Range (ACR): Standard PPO employs a clipping mechanism to ensure stable updates. In QuRL, the rollout is inherently "off-policy" because it is performed by a quantized model. ACR dynamically adjusts clipping bounds based on the divergence between the quantized actor and the full-precision actor, preventing training collapse during extended runs.
  • Update-Aware Quantization (UAQ): To mitigate the weight update problem, QuRL employs invariant scaling. By strategically scaling weights down and activations up before training, we amplify the magnitude of weight updates relative to quantization noise, ensuring that even minute RL updates are captured by the quantized model.

Experimental Setup

We evaluated QuRL on several demanding mathematical reasoning benchmarks, including GSM8K, AIME 2024, and the DeepScaleR Benchmark (comprising AMC, MATH, Minerva, and Olympiad Bench). Tests were conducted across model sizes ranging from 7B to 32B parameters using the vLLM inference engine for the rollout phase.

Key Results and Stability

QuRL demonstrates significant improvements in both training speed and model performance:

  • Throughput Gains: QuRL achieved 20% to 80% faster throughput during the rollout phase. The most substantial gains were observed in larger models (e.g., 32B), which are more heavily bottlenecked by computation.
  • Improved Accuracy: On the DeepScaleR benchmark, INT8 QuRL achieved a 55.5% average accuracy, outperforming the full-precision (BF16) baseline by 1.7%.
  • Training Stability: As shown in the training dynamics, naive quantization often leads to collapse (where rewards drop to zero). QuRL remains stable throughout the entire training cycle by managing the token clipping fraction and maintaining reward growth.
Comparison of mean training rewards and token clipped fractions across different quantization strategies.

Conclusion

QuRL proves that quantization can be more than a post-training compression step; it can be a core component of training infrastructure to bypass the autoregressive bottleneck. By treating the quantized actor as a "behavior policy" and addressing the weight update problem through invariant scaling, we enable faster training without sacrificing—and sometimes even improving—model accuracy. Future work will focus on optimizing KV cache quantization and expanding the scope to non-verifiable RLHF tasks.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.
BibTeX
@inproceedings{li2026qurl,
  author = {Yuhang Li and Reena Elangovan and Xin Dong and Priyadarshini Panda and Brucek Khailany},
  title = {{QuRL: Efficient Reinforcement Learning with Quantized Rollout}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=eG0bpCwdKn},
}
Figure