Abstract
Proximal Policy Optimization (PPO) has long served as a cornerstone of deep reinforcement learning (RL), utilizing an actor-critic architecture to stabilize training. However, the reliance on a separate value network (the critic) introduces significant memory overhead and approximation errors. We present Group Policy Gradient (GPG), a novel "critic-free" algorithm that eliminates the value network by employing group-based Monte Carlo estimates for advantage calculation. Inspired by the success of Group Relative Policy Optimization (GRPO) in large language model (LLM) alignment, GPG generalizes this approach to general Markov Decision Processes (MDPs). Our evaluations across standard Gymnasium benchmarks demonstrate that GPG matches or outperforms PPO in high-parallelism settings while significantly improving computational and memory efficiency.
Introduction
The industry standard for deep RL, Proximal Policy Optimization (PPO), relies on an actor-critic framework where a critic network estimates state values to reduce variance in policy updates. While effective, critics double memory requirements and add complexity through additional hyperparameters and potential approximation biases. Recent advancements in LLM training, specifically DeepSeek's GRPO, demonstrated that the critic could be replaced by averaging rewards across a group of parallel outputs. This paper explores whether this "critic-free" paradigm can be successfully applied to general RL tasks, such as robotics and control, where state-space dynamics are more complex than text generation.
Methods
The core innovation of Group Policy Gradient (GPG) is the replacement of the learned value function with a group-based estimator. GPG retains the stable "clipped objective" mechanism of PPO but discards the value network entirely.
Advantage Estimation via Grouping
Instead of querying a critic for the value of a state $s$, GPG utilizes $N$ parallel trajectories (rollouts). The advantage of an action is estimated by comparing its return to the average return of other trajectories within the same "group." To define these groups, the authors introduce binning functions $f(s, t)$ that identify "similar" states:
- Time-based binning: Compares rewards at the same timestep $t$, which proved to be a robust default for general RL.
- Spatial-based binning: Discretizes the state space to compare agents in similar regions.
- Universal binning: A simple baseline comparing all trajectories regardless of state.
The paper provides theoretical proof that the GPG estimator is consistent, converging to the true policy gradient as the group size increases. This avoids the systematic bias often introduced by neural network-based value approximations.
Experimental Setup
GPG was evaluated against PPO on four standard benchmarks from the OpenAI Gymnasium library:
- CartPole: Classic discrete control.
- CliffWalker: Grid-world navigation.
- LunarLander: Mixed discrete/continuous control.
- MuJoCo HalfCheetah: High-dimensional continuous robotics control.
The experiments focused on varying the number of parallel environments (1, 4, 16, and 128) to assess how group size impacts learning stability and performance.
Key Results
The empirical results highlight a clear trend: GPG's performance is highly scalable with parallelism.
Dominance at Scale
When utilizing 128 parallel environments, GPG consistently outperformed PPO across all benchmarks. In the complex HalfCheetah environment, GPG achieved a mean reward of 2773.61, nearly doubling PPO’s 1516.90 under identical parallel conditions. This suggests that the Monte Carlo-based advantage estimate, while noisier at low counts, becomes superior to a learned critic as more data points are available to compute the group mean.
Efficiency and Convergence
GPG offers significant memory efficiency by removing the critic's backpropagation pass. This reduction in VRAM usage is critical for training large-scale models where memory is the primary bottleneck. Furthermore, GPG demonstrated more stable convergence in high-dimensional tasks like HalfCheetah, likely due to the lack of approximation bias inherent in learned value functions.
Sample Efficiency Trade-offs
While GPG excels in iteration-based learning, it can be less sample-efficient than PPO in low-parallelism scenarios (e.g., 1 or 4 environments). In these cases, the group-based mean is highly sensitive to outliers, making the learned critic a more stable baseline for advantage estimation.
Conclusion and Future Work
Group Policy Gradient represents a significant step toward simplifying RL architectures. By leveraging parallelism as a mathematical tool for variance reduction rather than just a speedup, GPG eliminates the need for complex critic networks. While it requires higher parallelism to be effective, its efficiency and performance at scale make it a compelling alternative for modern RL applications. Future research will focus on adaptive binning functions to further improve sample efficiency in non-stationary environments.
Summary updated