TL;DR
This paper introduces TA-GRPO, a novel approach to training large language models (LLMs) for complex reasoning tasks. By rephrasing training questions into multiple equivalent versions, TA-GRPO effectively combats issues like gradient vanishing and diversity collapse, leading to significantly improved performance on mathematical reasoning benchmarks.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being used for complex reasoning tasks, particularly in mathematics. A popular training method for these models is Group Relative Policy Optimization (GRPO), which relies on rewards from multiple generated responses to a single question. However, GRPO suffers from two critical limitations:
- Gradient Vanishing: When a question is too easy or too hard, all generated responses receive the same reward, resulting in zero gradients and no learning signal.
- Diversity Collapse: The model tends to converge on a single reasoning strategy, failing to explore a diverse range of problem-solving approaches.
These issues hinder the LLMs' ability to generalize and robustly solve problems, especially those that are slightly different from the training data.
Methods
The paper proposes Transformation-Augmented GRPO (TA-GRPO), a method that enhances GRPO by augmenting the training data with rephrased questions. Here's how it works:
- Question Rephrasing: For each training question, TA-GRPO automatically generates multiple "problem-equivalent" rephrasings. These rephrasings alter wording, format, or information order while preserving the core mathematical problem. This is achieved using a large language model like GPT-4-Turbo.
- Expanded Response Group: Instead of generating responses to just the original question, TA-GRPO generates responses to the original question and all its rephrased versions. This creates a larger group of responses.
- Joint Group-Relative Normalization: Advantages are computed by pooling rewards across this entire expanded group. This means that even if the original question yields uniform rewards, the rephrased versions are likely to produce varied rewards, thus mitigating gradient vanishing.
- Anchored Importance Ratios: Crucially, the model updates are still anchored to the original question. This ensures that the learning signal is directed towards solving the intended problem, while the diverse rephrasings encourage exploration of different reasoning paths, combating diversity collapse.
Datasets
- Training Data: The primary dataset used for training is the MATH dataset [14], which contains 7498 high school-level mathematics questions.
- Evaluation Benchmarks:
- Competition-level mathematics: AMC [15], OlympiadBench [16], AIME24, and AIME25 [15].
- Out-of-distribution evaluation: Minerva [17] and GPQA-Diamond [18].
- Additional Data for Scaling Experiments: The OpenR1-MATH-220k dataset [37] was used to simulate training on larger datasets.
Key Insights and Ideas
- Rephrasing as a "Difficulty Shifter": The core idea is that rephrasing a question, even while preserving its meaning, can subtly alter the model's perceived difficulty. This shift is key to generating varied rewards and thus a useful training signal.
- Joint Normalization is Crucial: Pooling responses and normalizing rewards across the original and rephrased questions is more robust to gradient vanishing than normalizing per rephrasing. This is because it requires all (N+1) * G responses to have the same reward to fail, which is much less likely.
- Diversity Collapse is a Persistent Problem: The paper highlights that standard GRPO's failure modes (gradient vanishing and diversity collapse) intensify during training, making the proposed solution increasingly important as training progresses.
- Data Efficiency: TA-GRPO offers a way to achieve better performance without simply throwing more data at the problem, making it a more economical approach.
Key Results
- Consistent Pass@k Improvements: TA-GRPO consistently improves pass@k scores across four different LLMs (Qwen3-1.7B, Qwen3-4B, Llama-3.2-1B, Llama-3.2-3B) and all six tested benchmarks.
- Significant Gains: Average pass@32 gains over standard GRPO reached up to 5.09 points on competition math and 4.74 points on out-of-distribution benchmarks for the Qwen3 models. Peak gains were observed on specific benchmarks, e.g., 9.98 points on AIME24 for Qwen3-4B.
- Data Efficiency: TA-GRPO with the original training data achieved pass@32 scores comparable to baselines trained on up to 2.5x more data.
- Mechanism Validation: Experiments confirmed that rephrasing broadens the reward distribution and increases reasoning diversity, directly addressing the identified GRPO failure modes.
Limitations and Future Work
- Reliance on GPT-4-Turbo: The method currently relies on GPT-4-Turbo for question rephrasing, which incurs API costs. Future work could explore using comparable open-weight models.
- Training Time Overhead: TA-GRPO introduces additional overhead in training time due to generating and processing responses from rephrased questions.
- Input Distribution as a Lever: The paper suggests that the input distribution (how questions are presented) is a third important axis for RL post-training, alongside the loss and reward axes. Combining TA-GRPO with other GRPO advancements is a natural next step.
Summary updated