When Rewards Depend on the Group, GRPO Plays a Tournament

Junyi Gao, Yu Shi, Pingzhao Hu, Ewen M HarrisonarXiv 2026 · arXiv

In the rapidly evolving world of Reinforcement Learning from Human Feedback (RLHF), Group-Relative Policy Optimization (GRPO) has become a go-to algorithm, popularized by high-profile models like DeepSeek-R1. Its primary appeal is efficiency: by scoring samples against their group mean, it eliminates the need for a costly "critic" model.

However, a new paper titled "When Rewards Depend on the Group, GRPO Plays a Tournament" reveals a critical flaw. When we try to reward models for being diverse or matching a distribution, GRPO’s "efficiency" backfires, turning what should be a team effort into a zero-sum tournament that kills diversity.

TL;DR

Standard GRPO fails when rewards depend on the whole group (e.g., diversity bonuses) because its baseline turns optimization into a "tournament" where samples compete to beat their groupmates. The authors introduce Team-GRPO, a simple, unbiased estimator that restores cooperation and prevents diversity collapse without requiring extra model calls.

What Problem Does This Paper Address?

Modern RLHF isn't just about finding the "one true answer." We often want models to be diverse, to cover multiple valid reasoning paths, or to represent a broad distribution of human opinions. To achieve this, researchers use group-dependent rewards, where the score of one response depends on the others in its batch (e.g., "be good, but don't just repeat what the other samples said").

The problem is that GRPO was designed under the assumption that rewards are independent. When rewards are group-dependent, the group-relative baseline (subtracting the mean reward of groupmates) "leaks" information about a sample's effect on its peers. Instead of rewarding a sample for adding value to the group, GRPO rewards it for beating the group mean. As shown in Figure 1, this turns groupmates into rivals.

Figure 1. The problem. An answer helps its group by being good and by being different. GRPO credits each sample with how far it beats the group mean, so groupmates become rivals and converge on one answer. Team-GRPO credits each sample with what it adds to the group, so the group stays diverse.

Methods

The authors provide a rigorous theoretical diagnosis called the $\alpha$-identity. They prove that the expected update for these algorithms follows a specific form: $$\text{Update} \propto \nabla (\text{own reward}) + \alpha \cdot \nabla (\text{externality})$$ Where "externality" is the effect a sample has on its groupmates' rewards.

  • The True Gradient requires $\alpha = +1$ (you want to maximize the total group reward).
  • GRPO results in $\alpha = -1/(K-1)$ for a group of size $K$.

This negative $\alpha$ means GRPO actually penalizes a sample for helping its groupmates. To fix this, the authors propose Team-GRPO. It uses a "difference reward" approach—common in multi-agent RL—where each sample is credited with its marginal contribution to the group's total reward.

Crucially, for common rewards like pairwise diversity, Team-GRPO is computationally "free." It requires no extra calls to reward models or embeddings; it just uses the data already computed for the group reward.

Datasets

The researchers validated their theory and method across several settings:

  • A 9B Model Sandbox: Using Qwen3.5-9B on simple 4-option prompts to verify that training converges exactly where the theory predicts.
  • GlobalOpinionQA: A dataset of survey questions used to test "pluralistic alignment"—training a model to match the distribution of human opinions across different countries.
  • NoveltyBench: A benchmark for evaluating diversity and "set utility" in open-ended generation.
  • WildChat: Used for training data in the open-ended generation experiments.

Key Insights and Ideas

  • The "Tournament" Effect: Because GRPO has a negative $\alpha$, it rewards "spite." A sample can increase its advantage not just by being better, but by making its groupmates look worse.
  • The Attenuation Ladder: The paper introduces the "attenuation ladder" (Figure 5a), showing that GRPO is mathematically blind to a large portion of the diversity signal. At a group size of $K=2$, GRPO is 100% blind to pairwise diversity bonuses. Even as $K$ grows, it never captures more than half of the intended signal.
Figure 5a. The Attenuation Ladder. GRPO captures only a fraction of the true gradient for m-body interactions, and zero at K=2.
  • Diversity Collapse is Predictable: The authors could predict the final "concentration" of a 9B model's policy with a Pearson correlation of $r=0.995$. This suggests that diversity collapse isn't a random failure of RL, but a mathematical certainty when using the wrong baseline.

Key Results

  • Distributional Alignment: On GlobalOpinionQA, GRPO collapsed onto the majority answer, actually moving further away from the true human opinion distribution than the base model (error rose from 0.20 to 0.49). Team-GRPO successfully matched the distribution, cutting error to 0.08.
  • Open-Ended Generation: On NoveltyBench, GRPO suffered from "diversity collapse," producing repetitive, high-quality but low-utility sets of answers. Team-GRPO maintained diversity and significantly improved "set utility" (a gain of +1.16 over GRPO at $K=2$).
  • Efficiency at Small $K$: Team-GRPO works perfectly with just two samples per prompt ($K=2$), a regime where GRPO fails completely for diversity tasks. This allows for much more efficient training.

Limitations and Future Work

  • Variance: While Team-GRPO is unbiased, it has higher variance than GRPO or RLOO because it doesn't "cancel out" the externality signal. In practice, this is managed by using more prompts per batch.
  • Exchangeability: The current theory assumes "exchangeable" rewards (where the order of samples in the group doesn't matter). While this covers most current diversity methods, future work could extend this to non-exchangeable settings.
  • Beyond Diversity: The authors suggest that this "team vs. tournament" framework might apply to other areas of RLHF, such as multi-step reasoning or collaborative agent training.

Summary based on "When Rewards Depend on the Group, GRPO Plays a Tournament" (arXiv:2611.02404v1).

Summary updated

Abstract

Rewards for diversity, coverage and distribution matching make the reward of each sampled response depend on the rest of its group, and they are now routinely optimized with GRPO and RLOO. We prove that in this setting the group-relative baseline changes the objective. The expected update equals each response's own reward gradient plus $α$ times its effect on its groupmates' rewards, with $α=-1/(K-1)$ for groups of size $K$, so GRPO trains every response to beat its groupmates in a zero-sum tournament. It cannot see a pairwise diversity bonus when K=2, sees less than half of it at any K, and often does worse than using no baseline at all. Setting $α=+1$ gives Team-GRPO, an unbiased estimator for any group-dependent reward that works with two samples per prompt and requires no extra model calls for pairwise rewards. The theory predicts training quantitatively. It locates the fixed points of a 9B policy to within 0.02 in $\ell_1$ and predicts how concentrated trained policies become with Pearson r=0.995. On GlobalOpinionQA, GRPO collapses onto the majority answer and ends further from human opinion than the untrained model, while Team-GRPO cuts the error from $0.20$ to $0.08$. On NoveltyBench, Team-GRPO avoids the diversity collapse of GRPO and significantly improves set utility.
BibTeX
@misc{gao2026when,
  author = {Junyi Gao and Yu Shi and Pingzhao Hu and Ewen M Harrison},
  title = {{When Rewards Depend on the Group, GRPO Plays a Tournament}},
  year = {2026},
  eprint = {2611.02404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02404},
}
Figure