TL;DR
CorrGRPO is a refined version of Group Relative Policy Optimization (GRPO) designed for multi-reward reinforcement learning. By normalizing advantage estimates using reward correlations instead of raw covariances, it prevents large-scale rewards from drowning out smaller, informative ones, leading to significant performance gains in coding, tool use, and agent security.
What Problem Does This Paper Address?
Reinforcement Learning from Human Feedback (RLHF) often involves balancing multiple objectives. For example, when training a coding model, you might want code that is both correct (binary pass/fail) and efficient (execution time).
Standard GRPO computes "advantages"—the signal used to update the model—by centering rewards and dividing by their group standard deviation. When multiple rewards are summed, this denominator becomes the square root of the sum of all entries in the reward covariance matrix.
The authors identify a critical flaw: scale imbalance. If one reward component has a much larger numerical range or variance than others, it dominates the covariance sum. This effectively "mutes" the learning signal from smaller-scale rewards, even if they contain vital information for the task.
Methods
The authors propose Correlation-Normalized GRPO (CorrGRPO). The core idea is to decouple reward dependency from reward scale.
- Standard GRPO Denominator: Uses the sum of all pairwise covariances: $\sqrt{\sum \sum \text{Cov}(R_l, R_m) + \epsilon}$.
- CorrGRPO Denominator: Replaces covariances with Pearson correlation coefficients ($\rho$): $\sqrt{\sum \sum \hat{\rho}_{lm} + \epsilon}$.
By using correlations, every reward component contributes exactly "1" to the diagonal of the normalization matrix, regardless of its original scale. Off-diagonal terms now reflect the pure statistical relationship (reinforcement or tradeoff) between rewards without being skewed by their magnitudes. Crucially, CorrGRPO keeps the numerator (the centered total reward) unchanged, ensuring that the relative weights assigned to different rewards by the developer are still respected.
Datasets
The researchers evaluated CorrGRPO across three diverse domains using models ranging from 0.5B to 8B parameters (primarily the Qwen series):
- Coding Reasoning: Trained on LeetCodeDataset and evaluated on HumanEval, MBPP, and LiveCodeBench v6. Rewards included syntax validity, compilation success, test pass rate, and execution efficiency.
- Tool Calling: Trained on RLLA-4K and evaluated on API-Bank. Rewards focused on function name, parameter name, and parameter value accuracy.
- Agent Utility vs. Security: Trained on AgentDojo and evaluated on Agent Security Bench (ASB) and InjecAgent. This task required balancing task completion (utility) with resistance to prompt injections (security).
Key Insights and Ideas
- The Scale Imbalance Trap: The paper provides a striking example (Figure 2 in the paper) where a single large-scale reward with weak correlations accounts for 91% of the normalization denominator in standard GRPO, effectively suppressing the signals from two highly correlated but smaller-scale rewards.
- Implicit Scaling: GRPO naturally scales updates: positive correlations (redundant info) reduce update magnitude, while negative correlations (tradeoffs) increase it. CorrGRPO preserves this elegant property while making it "scale-blind."
- Entropy Preservation: Interestingly, CorrGRPO was found to maintain higher policy entropy throughout training compared to GRPO. This suggests it prevents the model from collapsing onto a single strategy too early, allowing for better exploration of the multi-reward landscape.
Key Results
CorrGRPO consistently outperformed GRPO and other recent variants like GDPO across all benchmarks:
- Coding: On the LeetCodeDataset, Qwen2.5-Coder-7B-Instruct saw its Pass@1 jump from 15.79% (GRPO) to 24.12% (CorrGRPO). It also pushed the Pareto frontier, achieving better efficiency without sacrificing accuracy.
- Tool Calling: Across three different backbones, CorrGRPO achieved the highest average accuracy, showing particular strength in complex parameter-value generation.
- Agent Security: In the AgentDojo environment, CorrGRPO significantly improved "Joint Accuracy" (a metric balancing utility and safety). For a 3B model, this rose from 60.13% to 89.62%.
Limitations and Future Work
- Reward Specification: While CorrGRPO balances the learning signal, it cannot fix fundamentally biased or poorly designed reward functions. If the rewards themselves don't capture the desired behavior, the model will still fail to align.
- Computational Overhead: Calculating the correlation matrix adds a small amount of compute per batch, though the authors note this is negligible compared to the cost of model forward/backward passes.
- Future Directions: The authors suggest exploring how this correlation-based approach could be combined with dynamic reward weighting schemes that adjust the numerator weights during training.
Not discussed in the paper: The impact of extremely small group sizes (e.g., $n=2$) on the stability of correlation estimates.
Summary updated