GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

Jixiao Zhang, Chunsheng ZuoEMNLP 2025

TL;DR

This paper introduces GRPO-LEAD, a novel reinforcement learning framework that significantly improves mathematical reasoning in large language models. By incorporating length-regularized rewards, explicit penalties for incorrect answers, and difficulty-aware advantage reweighting, GRPO-LEAD achieves state-of-the-art performance, producing more accurate, concise, and efficient solutions.

What Problem Does This Paper Address?

Large language models (LLMs) have shown impressive capabilities in mathematical reasoning, often by generating step-by-step "chains of thought." Reinforcement learning (RL) fine-tuning, particularly using Group Relative Policy Optimization (GRPO), has been crucial for this progress. However, existing GRPO methods face several challenges:

  • Reward Sparsity: Binary accuracy metrics can lead to uniform reward signals, hindering effective learning.
  • Verbosity: Models tend to generate overly long, verbose solutions to maximize rewards, sacrificing conciseness and interpretability.
  • Lack of Precision: Current reward formulations don't explicitly penalize incorrect answers, encouraging guessing over rigorous reasoning.
  • Difficulty Neglect: Rewards are often applied uniformly, causing models to over-optimize simpler problems while neglecting harder ones.
  • Computational Cost: RL fine-tuning is resource-intensive, limiting accessibility and scalability.

GRPO-LEAD aims to address these limitations to create more robust, efficient, and accurate mathematical reasoning models.

Methods

GRPO-LEAD enhances the GRPO framework with three core innovations:

  1. Length-Dependent Accuracy Reward: This new reward function encourages conciseness by penalizing overly long correct answers. It dynamically calibrates rewards based on a question's specific distribution of response lengths and accuracy, moving beyond static length constraints.
  2. Explicit Penalty for Incorrect Answers: To improve precision, GRPO-LEAD introduces a direct penalty for incorrect solutions. This acts as a regularizer, forcing the model to be more conservative and less likely to guess, thereby increasing true accuracy.
  3. Difficulty-Aware Advantage Reweighting: This strategy addresses the issue of neglecting challenging problems. It dynamically adjusts the magnitude of policy updates based on problem difficulty, amplifying learning signals for harder tasks and re-anchoring the model's focus. This is achieved by reweighting advantages based on an empirical correctness ratio of responses for each question.

The paper also explores the impact of base model scale and supervised fine-tuning (SFT) on RL effectiveness.

Datasets

  • Training: The primary training data is sourced from the DEEPSCALER dataset, filtered for problems with difficulty ratings above 2.5 (approx. 9,000 questions). For stage 2 of the 14B model experiments, a refined dataset of about 2,283 questions was used, along with 3,524 challenging problems from the Light-R1 dataset.
  • Evaluation: The models were evaluated on two mathematical benchmarks: AIME24 and AIME25. An additional evaluation was performed on the LiveCodeBench benchmark.

Key Insights and Ideas

  • Conciseness as a Learning Signal: Penalizing overly verbose correct solutions doesn't just reduce output length; it actually accelerates early-stage convergence in RL training, suggesting it provides a more informative learning signal.
  • Difficulty Matters: Explicitly reweighting advantages to focus on harder problems significantly improves robustness and generalization, preventing models from becoming overly reliant on simpler tasks.
  • The Power of Explicit Penalties: Directly penalizing incorrect answers acts as a crucial "regularizer for thinking," discouraging hasty guesses and leading to more reliable reasoning, even if it requires slightly longer thought processes.
  • SFT + RL Synergy: Supervised fine-tuning (SFT) can improve reasoning quality and potentially increase output length, but subsequent RL fine-tuning (like GRPO-LEAD) can effectively mitigate these issues and lead to better overall performance.

Key Results

  • State-of-the-Art Performance: GRPO-LEAD achieved state-of-the-art results on AIME24 and AIME25 benchmarks for 14B-scale models, outperforming strong baselines like DeepSeek-14B and Light-R1-14B-DS.
  • Improved Accuracy: On AIME24, GRPO-LEAD achieved the highest Pass@1 (0.470) and Cons@32 (0.800). On AIME25, it achieved the best Pass@1 (0.539) and matched the best Cons@32 (0.767).
  • Reduced Verbosity: The length-dependent reward significantly reduced average response lengths, for example, by 24.5% and 26.8% on AIME24 and AIME25 respectively, compared to GRPO+L.
  • Enhanced Performance on Difficult Problems: Advantage reweighting showed a substantial performance gap increase (13.7% in Pass@1) for difficult problems compared to GRPO+L.
  • Efficiency: The proposed method, particularly LEAD-stage1, achieved competitive results with significantly less training time (approx. 24 hours on 8 H20 GPUs) compared to baselines.

Limitations and Future Work

  • Domain Specificity: The reported gains are specific to mathematical reasoning tasks. Further studies are needed to evaluate GRPO-LEAD's effectiveness on broader question-answering or logical reasoning domains where correctness signals and domain structures differ.
  • Limited Compute: The authors had limited compute resources, which prevented them from conducting more comprehensive experiments, such as providing the validation curve for the 7B model with an explicit penalty or reproducing comparisons with the original GRPO model.
  • Hyperparameter Tuning: A formal hyperparameter search was not performed to showcase the rationale behind the chosen hyperparameters for the modifications.
  • Future Work: The paper suggests that managing partial correctness and extending these techniques to broader domains are important future research directions.

Summary updated

BibTeX
@inproceedings{zhang2025grpo,
  author = {Jixiao Zhang and Chunsheng Zuo},
  title = {{GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure