CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning

Wenbin Hu, Huihao Jing, Haochen Shi, Yuxuan Liu, Haoran Li, Yangqiu SongarXiv 2026 · arXiv

TL;DR

CorrGRPO is a refined version of Group Relative Policy Optimization (GRPO) designed for multi-reward reinforcement learning. By normalizing advantage estimates using reward correlations instead of raw covariances, it prevents large-scale rewards from drowning out smaller, informative ones, leading to significant performance gains in coding, tool use, and agent security.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) often involves balancing multiple objectives. For example, when training a coding model, you might want code that is both correct (binary pass/fail) and efficient (execution time).

Standard GRPO computes "advantages"—the signal used to update the model—by centering rewards and dividing by their group standard deviation. When multiple rewards are summed, this denominator becomes the square root of the sum of all entries in the reward covariance matrix.

The authors identify a critical flaw: scale imbalance. If one reward component has a much larger numerical range or variance than others, it dominates the covariance sum. This effectively "mutes" the learning signal from smaller-scale rewards, even if they contain vital information for the task.

Methods

The authors propose Correlation-Normalized GRPO (CorrGRPO). The core idea is to decouple reward dependency from reward scale.

  1. Standard GRPO Denominator: Uses the sum of all pairwise covariances: $\sqrt{\sum \sum \text{Cov}(R_l, R_m) + \epsilon}$.
  2. CorrGRPO Denominator: Replaces covariances with Pearson correlation coefficients ($\rho$): $\sqrt{\sum \sum \hat{\rho}_{lm} + \epsilon}$.

By using correlations, every reward component contributes exactly "1" to the diagonal of the normalization matrix, regardless of its original scale. Off-diagonal terms now reflect the pure statistical relationship (reinforcement or tradeoff) between rewards without being skewed by their magnitudes. Crucially, CorrGRPO keeps the numerator (the centered total reward) unchanged, ensuring that the relative weights assigned to different rewards by the developer are still respected.

Figure 1: An overview of CorrGRPO. (1) Left: CorrGRPO replaces total-reward standard deviation normalization with a correlation-based denominator while retaining the centered total reward. (2) Right: Validation accuracy (top) and efficiency (bottom), measured by Mean@1, for Qwen2.5-Coder-7B-Instruct on LeetCodeDataset.

Datasets

The researchers evaluated CorrGRPO across three diverse domains using models ranging from 0.5B to 8B parameters (primarily the Qwen series):

  • Coding Reasoning: Trained on LeetCodeDataset and evaluated on HumanEval, MBPP, and LiveCodeBench v6. Rewards included syntax validity, compilation success, test pass rate, and execution efficiency.
  • Tool Calling: Trained on RLLA-4K and evaluated on API-Bank. Rewards focused on function name, parameter name, and parameter value accuracy.
  • Agent Utility vs. Security: Trained on AgentDojo and evaluated on Agent Security Bench (ASB) and InjecAgent. This task required balancing task completion (utility) with resistance to prompt injections (security).

Key Insights and Ideas

  • The Scale Imbalance Trap: The paper provides a striking example (Figure 2 in the paper) where a single large-scale reward with weak correlations accounts for 91% of the normalization denominator in standard GRPO, effectively suppressing the signals from two highly correlated but smaller-scale rewards.
  • Implicit Scaling: GRPO naturally scales updates: positive correlations (redundant info) reduce update magnitude, while negative correlations (tradeoffs) increase it. CorrGRPO preserves this elegant property while making it "scale-blind."
  • Entropy Preservation: Interestingly, CorrGRPO was found to maintain higher policy entropy throughout training compared to GRPO. This suggests it prevents the model from collapsing onto a single strategy too early, allowing for better exploration of the multi-reward landscape.
A toy example illustrating how a large-scale reward (r3) dominates the covariance matrix in GRPO, whereas CorrGRPO balances the influence using the correlation matrix.

Key Results

CorrGRPO consistently outperformed GRPO and other recent variants like GDPO across all benchmarks:

  • Coding: On the LeetCodeDataset, Qwen2.5-Coder-7B-Instruct saw its Pass@1 jump from 15.79% (GRPO) to 24.12% (CorrGRPO). It also pushed the Pareto frontier, achieving better efficiency without sacrificing accuracy.
  • Tool Calling: Across three different backbones, CorrGRPO achieved the highest average accuracy, showing particular strength in complex parameter-value generation.
  • Agent Security: In the AgentDojo environment, CorrGRPO significantly improved "Joint Accuracy" (a metric balancing utility and safety). For a 3B model, this rose from 60.13% to 89.62%.

Limitations and Future Work

  • Reward Specification: While CorrGRPO balances the learning signal, it cannot fix fundamentally biased or poorly designed reward functions. If the rewards themselves don't capture the desired behavior, the model will still fail to align.
  • Computational Overhead: Calculating the correlation matrix adds a small amount of compute per batch, though the authors note this is negligible compared to the cost of model forward/backward passes.
  • Future Directions: The authors suggest exploring how this correlation-based approach could be combined with dynamic reward weighting schemes that adjust the numerator weights during training.

Not discussed in the paper: The impact of extremely small group sizes (e.g., $n=2$) on the stability of correlation estimates.

Summary updated

Abstract

Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.
BibTeX
@misc{hu2026corrgrpo,
  author = {Wenbin Hu and Huihao Jing and Haochen Shi and Yuxuan Liu and Haoran Li and Yangqiu Song},
  title = {{CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning}},
  year = {2026},
  eprint = {2609.36820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.36820},
}
Figure