Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments

Bryan L. M. de Oliveira, Felipe Vieira Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. L. Soares, Luckeciano C. MeloarXiv 2025 · DOI · arXiv

Abstract

This paper presents a systematic evaluation of Group Relative Policy Optimization (GRPO)—a "critic-free" algorithm widely used in Large Language Model (LLM) alignment—within the context of classical reinforcement learning (RL) benchmarks. By removing the learned value function (critic), GRPO significantly reduces memory and computational overhead. However, our experiments across Gymnasium environments reveal that while GRPO can excel in simple tasks with early termination, it generally fails to match the performance and sample efficiency of Proximal Policy Optimization (PPO) in complex, long-horizon locomotion tasks. We conclude that while GRPO is a powerful tool for specific high-memory domains, the learned critic remains essential for general-purpose high-dimensional control.

Introduction

In modern Reinforcement Learning, the Actor-Critic framework, exemplified by Proximal Policy Optimization (PPO), is the industry standard. The "Critic" (a learned value function) estimates state values to reduce gradient variance for the "Actor" (the policy). Despite its success, the critic introduces substantial memory costs, which becomes a massive bottleneck when training giant models like LLMs.

Recently, GRPO emerged as a way to train models without a critic. Instead of learning a value function, it estimates "advantage" by comparing a group of outputs to one another—essentially grading on a curve. While this works for LLMs, it was previously unknown if this "critic-free" approach translates to classical control tasks like balancing a pole or robotic locomotion. This study investigates whether GRPO is a general-purpose RL breakthrough or a niche tool for language modeling.

Methodology

The researchers compared standard PPO against GRPO and several other "critic-free" baselines across a variety of control tasks.

The GRPO Approach

Instead of using a neural network to predict rewards ($V(s)$), GRPO takes a group of $G$ trajectories and calculates the mean and standard deviation of their returns. The "advantage" for a specific action is determined by how much better that trajectory was compared to the group average.

Grouping Strategy and Ablations

Since classical RL lacks the natural "prompts" found in LLMs, the authors formed groups by taking trajectories collected in parallel from different environment instances. They tested several factors affecting the learning process:

  • Discount Factor ($\gamma$): Testing values from 0.9 to 1.0.
  • Group Size ($G$): Evaluating whether larger groups provide better signals.
  • Baseline Types: Comparing exponential moving averages against batch means.

Experimental Setup

The study utilized the standard Gymnasium library, focusing on five classic benchmarks:

  1. Discrete Control: CartPole-v1 and Acrobot-v1.
  2. Continuous Control: MountainCarContinuous-v0, HalfCheetah-v4, and Humanoid-v4.

These environments represent a range of difficulties, from short-horizon "sanity checks" (CartPole) to high-dimensional locomotion (Humanoid).

Results and Analysis

The experimental results reveal a clear performance gap between actor-critic and critic-free methods in most scenarios.

Comparison of PPO and GRPO-Batch performance across CartPole-v1, Acrobot-v1, and HalfCheetah-v4.

PPO Dominance in Complexity

In 4 out of 5 environments, standard PPO with a learned critic significantly outperformed GRPO in terms of sample efficiency and final return. In high-dimensional tasks like Humanoid-v4 and HalfCheetah-v4, the lack of a learned critic made it difficult for the agent to receive the "step-by-step" guidance necessary for stable learning.

The CartPole Exception

Interestingly, GRPO outperformed PPO in CartPole-v1. The authors suggest this is because PPO's critic can lead to "overtraining" or policy collapse in very simple, short-horizon tasks, whereas GRPO’s higher variance acted as a form of beneficial regularization.

Insights on Environment Dynamics

GRPO performed best in environments with "early termination" (like CartPole, where the game ends if you fail). These clear boundaries create a sharp contrast between "good" and "bad" trajectories. In environments like HalfCheetah, which run for a fixed time, the learning signal becomes too diluted for GRPO to handle effectively without a critic.

Ablation studies on HalfCheetah-v4 demonstrating the sensitivity of critic-free methods to various hyperparameter and grouping settings.

As shown in Figure 6, HalfCheetah proved particularly challenging. The authors found that smaller group sizes ($G=8$) surprisingly outperformed larger ones ($G=128$), likely because larger groups introduced more noise when mixing unrelated parallel runs.

Conclusion

While GRPO offers significant memory savings by ditching the value function, it struggles to match the performance of PPO in complex, long-horizon tasks where a learned critic remains essential. The study suggests that GRPO's success in LLMs may be due to the specific nature of language tasks (e.g., prompt-based grouping) rather than general superiority over actor-critic methods. Future work could explore more sophisticated grouping strategies, such as clustering trajectories based on state similarity, to improve GRPO's performance in continuous control.

Summary updated

BibTeX
@misc{oliveira2025learning,
  author = {Bryan L. M. de Oliveira and Felipe Vieira Frujeri and Marcos P. C. M. Queiroz and Luana G. B. Martins and Telma W. L. Soares and Luckeciano C. Melo},
  title = {{Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments}},
  year = {2025},
  eprint = {2511.03527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.03527},
}
Figure