RL-finetuning LLMs from on- and off-policy data with a single algorithm

Yunhao Tang, Taco Cohen, David W. Zhang, Gabriel Synnaeve, Rémi Munos

TL;DR

Researchers from Meta have introduced AGRO (Any-Generation Reward Optimization), a unified reinforcement learning algorithm that fine-tunes Large Language Models (LLMs) using both online (on-policy) and offline (off-policy) data. By leveraging a principle called "generation consistency," AGRO achieves superior data efficiency and stability, outperforming standard RLHF baselines on complex mathematical reasoning tasks.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) is the standard for aligning LLMs, but it usually falls into two distinct camps:

  1. Online RL: The model generates new responses and gets feedback in real-time. This is effective but computationally expensive and requires a constant "reward model" loop.
  2. Offline RL: The model learns from a static, pre-collected dataset. This is cheaper but often fails to reach the same performance peaks as online methods.

The "holy grail" is an algorithm that works seamlessly across both. Currently, if you try to use a standard on-policy algorithm (like PPO) on offline data, the math breaks—the model fails to converge to the optimal policy because the data distribution doesn't match the model's current behavior. AGRO bridges this gap by providing a single mathematical framework that handles any data source.

Methods

The secret sauce of AGRO is Generation Consistency.

The authors prove a theoretical property: for an optimally tuned model, the "regularized reward" (the raw reward minus a penalty for straying too far from the starting model) should be constant for any response the model might generate for a given prompt. In other words, if the model is perfect, every valid path it takes should have the same "net value."

Based on this, AGRO works by:

  • Minimizing Variance: Instead of just trying to maximize reward, the algorithm minimizes the variance of the regularized reward across different generations.
  • Unified Gradient: They derive a gradient estimation that remains unbiased whether the data is "on-policy" (generated by the current model) or "off-policy" (from a fixed dataset).
  • Token-Level Optimization: To make it work for LLMs, they adapt the algorithm to operate at the token level, using variance reduction techniques to handle the long sequences typical of model responses.

Datasets

The authors evaluated AGRO using the MATH dataset, a challenging benchmark for mathematical reasoning.

  • Model: Llama 3 (8B parameters).
  • Training: 7,500 examples from the MATH training set.
  • Evaluation: 5,000 examples from the test set.
  • Reward Signal: A binary reward (1 for a correct answer, 0 for incorrect) determined by comparing the model's final answer to the ground truth using the Sympy library.

Key Insights and Ideas

  • The Consistency "Aha!" Moment: The realization that optimality implies consistency across generations allows the researchers to move away from complex importance sampling (which often plagues off-policy RL) and toward a simpler variance-minimization objective.
  • Breaking the Dichotomy: AGRO shows that the divide between online and offline RL is somewhat artificial. By focusing on the properties of the optimal policy rather than the current sampling distribution, you can create a more robust learner.
  • Data Efficiency: In on-policy tests, AGRO was significantly more "sample efficient," meaning it learned faster and reached higher performance levels with the same amount of data compared to standard policy gradient methods like RLOO.

Key Results

  • On-Policy Superiority: On the MATH dataset, AGRO achieved a +7% performance lift over the reference model, while standard RL baselines only managed a +4% lift.
  • Off-Policy Stability: In the offline setting (learning from a static dataset), standard RL algorithms failed to converge to the optimal policy. AGRO, however, successfully converged, demonstrating its robustness for "offline-to-online" transitions.
  • KL-Efficiency: AGRO proved to be highly "KL-efficient," meaning it achieved better performance gains while staying closer to the original model's distribution, which helps prevent "model collapse" or nonsensical outputs.

Limitations and Future Work

  • Extreme Off-Policy Instability: While AGRO is more stable than its predecessors, the authors note that learning can still become unstable if the offline data is extremely far removed from what the model would naturally generate.
  • Implementation Complexity: The token-level variance reduction (while effective) is mathematically sophisticated and requires careful implementation to avoid overhead in large-scale training.
  • Broader Generalization: The study focused heavily on mathematical reasoning (where rewards are clear and binary). Future work is needed to see how AGRO handles more subjective tasks like creative writing or general-purpose chat, where reward signals are noisier.

Summary updated

Abstract

We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for finetuning Large Language Models. AGRO leverages the concept of response consistency, which states that the optimal policy satisfies a notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the MATH dataset over baseline methods.
BibTeX
@inproceedings{tang2026rl,
  author = {Yunhao Tang and Taco Cohen and David W. Zhang and Gabriel Synnaeve and Rémi Munos},
  title = {{RL-finetuning LLMs from on- and off-policy data with a single algorithm}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=v8n0UHCu8R},
}

Loading related content...

Figure