Abstract
This paper presents a systematic evaluation of Group Relative Policy Optimization (GRPO)—a "critic-free" algorithm widely used in Large Language Model (LLM) alignment—within the context of classical reinforcement learning (RL) benchmarks. By removing the learned value function (critic), GRPO significantly reduces memory and computational overhead. However, our experiments across Gymnasium environments reveal that while GRPO can excel in simple tasks with early termination, it generally fails to match the performance and sample efficiency of Proximal Policy Optimization (PPO) in complex, long-horizon locomotion tasks. We conclude that while GRPO is a powerful tool for specific high-memory domains, the learned critic remains essential for general-purpose high-dimensional control.
Introduction
In modern Reinforcement Learning, the Actor-Critic framework, exemplified by Proximal Policy Optimization (PPO), is the industry standard. The "Critic" (a learned value function) estimates state values to reduce gradient variance for the "Actor" (the policy). Despite its success, the critic introduces substantial memory costs, which becomes a massive bottleneck when training giant models like LLMs.
Recently, GRPO emerged as a way to train models without a critic. Instead of learning a value function, it estimates "advantage" by comparing a group of outputs to one another—essentially grading on a curve. While this works for LLMs, it was previously unknown if this "critic-free" approach translates to classical control tasks like balancing a pole or robotic locomotion. This study investigates whether GRPO is a general-purpose RL breakthrough or a niche tool for language modeling.
Methodology
The researchers compared standard PPO against GRPO and several other "critic-free" baselines across a variety of control tasks.
The GRPO Approach
Instead of using a neural network to predict rewards ($V(s)$), GRPO takes a group of $G$ trajectories and calculates the mean and standard deviation of their returns. The "advantage" for a specific action is determined by how much better that trajectory was compared to the group average.
Grouping Strategy and Ablations
Since classical RL lacks the natural "prompts" found in LLMs, the authors formed groups by taking trajectories collected in parallel from different environment instances. They tested several factors affecting the learning process:
- Discount Factor ($\gamma$): Testing values from 0.9 to 1.0.
- Group Size ($G$): Evaluating whether larger groups provide better signals.
- Baseline Types: Comparing exponential moving averages against batch means.
Experimental Setup
The study utilized the standard Gymnasium library, focusing on five classic benchmarks:
- Discrete Control: CartPole-v1 and Acrobot-v1.
- Continuous Control: MountainCarContinuous-v0, HalfCheetah-v4, and Humanoid-v4.
These environments represent a range of difficulties, from short-horizon "sanity checks" (CartPole) to high-dimensional locomotion (Humanoid).
Results and Analysis
The experimental results reveal a clear performance gap between actor-critic and critic-free methods in most scenarios.
PPO Dominance in Complexity
In 4 out of 5 environments, standard PPO with a learned critic significantly outperformed GRPO in terms of sample efficiency and final return. In high-dimensional tasks like Humanoid-v4 and HalfCheetah-v4, the lack of a learned critic made it difficult for the agent to receive the "step-by-step" guidance necessary for stable learning.
The CartPole Exception
Interestingly, GRPO outperformed PPO in CartPole-v1. The authors suggest this is because PPO's critic can lead to "overtraining" or policy collapse in very simple, short-horizon tasks, whereas GRPO’s higher variance acted as a form of beneficial regularization.
Insights on Environment Dynamics
GRPO performed best in environments with "early termination" (like CartPole, where the game ends if you fail). These clear boundaries create a sharp contrast between "good" and "bad" trajectories. In environments like HalfCheetah, which run for a fixed time, the learning signal becomes too diluted for GRPO to handle effectively without a critic.
As shown in Figure 6, HalfCheetah proved particularly challenging. The authors found that smaller group sizes ($G=8$) surprisingly outperformed larger ones ($G=128$), likely because larger groups introduced more noise when mixing unrelated parallel runs.
Conclusion
While GRPO offers significant memory savings by ditching the value function, it struggles to match the performance of PPO in complex, long-horizon tasks where a learned critic remains essential. The study suggests that GRPO's success in LLMs may be due to the specific nature of language tasks (e.g., prompt-based grouping) rather than general superiority over actor-critic methods. Future work could explore more sophisticated grouping strategies, such as clustering trajectories based on state similarity, to improve GRPO's performance in continuous control.
Summary updated