Abstract
We introduce a novel reinforcement learning algorithm, AGRO (Any-Generation Reward Optimization), for fine-tuning large language models (LLMs). AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on mathematical reasoning datasets over baseline algorithms.
Introduction
Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs) with human preferences. However, current methodologies typically bifurcate into two distinct paradigms: online RL, which is effective but computationally expensive due to real-time feedback loops, and offline RL, which is more efficient but often struggles to reach peak performance. Researchers from Meta have introduced AGRO (Any-Generation Reward Optimization) to bridge this gap. AGRO is a unified framework that enables fine-tuning using both online (on-policy) and offline (off-policy) data with superior efficiency and stability.
Methodology: Generation Consistency
The core innovation of AGRO is the principle of Generation Consistency. Theoretically, for an optimally tuned model, the regularized reward (the raw reward minus a penalty for deviating from the reference model) should remain constant across any response the model generates for a specific prompt.
Based on this insight, AGRO operates by:
- Minimizing Variance: The algorithm seeks to minimize the variance of the regularized reward across different generations rather than solely maximizing raw reward.
- Unified Gradient Estimation: AGRO utilizes a gradient estimator that remains unbiased regardless of whether the data source is on-policy (current model generations) or off-policy (static datasets).
- Token-Level Optimization: The framework is adapted for the long sequences of LLMs by operating at the token level and employing variance reduction techniques to maintain stability.
Experimental Setup
The authors evaluated AGRO using the MATH dataset, a rigorous benchmark for mathematical reasoning. The experiments utilized a Llama 3 (8B) model trained on 7,500 examples from the MATH training set. Performance was measured against a test set of 5,000 examples, with success defined by a binary reward (correct/incorrect) verified via the Sympy library.
Key Results
AGRO demonstrates significant improvements in both learning efficiency and final model performance. In on-policy evaluations on the MATH dataset, AGRO achieved a +7% performance lift over the reference model, notably outperforming standard RL baselines which only achieved a +4% lift.
Furthermore, AGRO exhibits high KL-efficiency. It achieves greater performance gains while maintaining a smaller Kullback–Leibler (KL) divergence from the original model. This characteristic is crucial for preventing "model collapse" and ensuring the model retains its general capabilities while specializing in reasoning tasks.
Discussion and Insights
AGRO's success suggests that the traditional dichotomy between online and offline RL is somewhat artificial. By focusing on the properties of the optimal policy—specifically generation consistency—rather than the sampling distribution, AGRO provides a more robust learning signal.
While AGRO shows remarkable stability, the authors note that extreme distribution shifts in offline data can still introduce instability. Future research will focus on extending AGRO to more subjective domains, such as creative writing or general-purpose dialogue, where reward signals are inherently noisier than the binary outcomes found in mathematical reasoning.
Summary updated