On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

Wenlong Deng, Yi Ren, Muchen Li, Danica J. Sutherland, Xiaoxiao Li, Christos Thrampoulidis

Abstract

Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm. Despite its adoption, we identify a previously unrecognized phenomenon termed Lazy Likelihood Displacement (LLD), where the likelihood of correct responses marginally increases or even decreases during training. This behavior is attributed to the influence of negative gradients. We provide a theoretical analysis and introduce Negative Token Hidden Reward (NTHR), a method that downweights penalties on tokens contributing to LLD by using correct responses as anchors. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across various model sizes.

Introduction: The Lazy Likelihood Displacement (LLD) Problem

Reinforcement Learning (RL) is a critical component in developing reasoning capabilities in Large Language Models (LLMs). Group Relative Policy Optimization (GRPO) has gained traction due to its efficiency, as it eliminates the need for a separate value model by comparing responses within a group. However, a significant misalignment issue exists within this framework.

We identify a phenomenon called Lazy Likelihood Displacement (LLD). In standard GRPO, when a model generates an incorrect response, the algorithm penalizes every token in that response equally. Because incorrect responses often share high semantic similarity with correct ones—containing valid reasoning steps followed by minor errors—this broad penalization inadvertently suppresses "good" reasoning tokens that also appear in correct answers. Consequently, the model's confidence in correct answers may stagnate or even decline during training.

The LLD phenomenon across different models. GRPO (blue) often shows lower likelihood increases for correct samples compared to positive-only training.

Methodology: Negative Token Hidden Reward (NTHR)

To address LLD, we propose Negative Token Hidden Reward (NTHR). This method performs "token-level surgery" to selectively reduce penalties on tokens in incorrect responses that are semantically aligned with correct reasoning.

NTHR operates through three main steps:

  1. Anchoring: It uses correct responses within a group as anchors to define "good" reasoning.
  2. Influence Calculation: It measures the influence of each token in an incorrect response on the likelihood of the correct responses using internal model embeddings.
  3. Selective Penalization: Tokens identified as highly similar to those in correct responses receive a reduced penalty. This prevents the model from "unlearning" valid mathematical concepts or logical steps just because they appeared in a failed attempt.
Qualitative examples of "Almost Correct" and "Format Issue" responses where broad penalization is counterproductive.

This approach is particularly effective for "almost correct" answers or those with simple formatting issues (e.g., missing a required tag), where the bulk of the generated text is actually high-quality reasoning.

Experimental Results

We evaluated NTHR against standard GRPO and "Positive Only" training across several mathematical reasoning benchmarks, including AIME 2024, AMC, and MATH500.

  • Performance Gains: NTHR consistently outperformed GRPO. For instance, on the Qwen2.5-1.5B-Instruct model, NTHR improved the average benchmark score from 26.96% to 28.48%.
  • Stability: While training on positive samples only ("Pos Only") can lead to higher likelihood increases, it often results in a performance drop (e.g., a 1.3% decrease in overall accuracy). NTHR maintains the necessary corrective signal of negative reinforcement while ensuring it is applied precisely.
  • Scalability: The benefits of NTHR were observed across models ranging from 0.5B to 3B parameters, suggesting the LLD phenomenon is a fundamental characteristic of group-based RL in LLMs.

Conclusion

The discovery of Lazy Likelihood Displacement highlights a subtle but impactful flaw in current RLHF practices. By introducing NTHR, we provide a mechanism to protect valuable reasoning steps within incorrect outputs. This surgical approach to negative gradients allows models to learn more efficiently from their mistakes, leading to more robust reasoning capabilities without the "lazy" learning artifacts seen in standard GRPO.

Summary updated

Abstract

Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO's learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.
BibTeX
@inproceedings{deng2025effect,
  author = {Wenlong Deng and Yi Ren and Muchen Li and Danica J. Sutherland and Xiaoxiao Li and Christos Thrampoulidis},
  title = {{On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=2K9QsDaqkM},
}
Figure