Transform-Augmented GRPO Improves Pass@k

Khiem Le, Youssef Mroueh, Phuc Nguyen, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla

TL;DR

This paper introduces TA-GRPO, a novel approach to training large language models (LLMs) for complex reasoning tasks. By rephrasing training questions into multiple equivalent versions, TA-GRPO effectively combats issues like gradient vanishing and diversity collapse, leading to significantly improved performance on mathematical reasoning benchmarks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being used for complex reasoning tasks, particularly in mathematics. A popular training method for these models is Group Relative Policy Optimization (GRPO), which relies on rewards from multiple generated responses to a single question. However, GRPO suffers from two critical limitations:

  1. Gradient Vanishing: When a question is too easy or too hard, all generated responses receive the same reward, resulting in zero gradients and no learning signal.
  2. Diversity Collapse: The model tends to converge on a single reasoning strategy, failing to explore a diverse range of problem-solving approaches.

These issues hinder the LLMs' ability to generalize and robustly solve problems, especially those that are slightly different from the training data.

Methods

The paper proposes Transformation-Augmented GRPO (TA-GRPO), a method that enhances GRPO by augmenting the training data with rephrased questions. Here's how it works:

  • Question Rephrasing: For each training question, TA-GRPO automatically generates multiple "problem-equivalent" rephrasings. These rephrasings alter wording, format, or information order while preserving the core mathematical problem. This is achieved using a large language model like GPT-4-Turbo.
  • Expanded Response Group: Instead of generating responses to just the original question, TA-GRPO generates responses to the original question and all its rephrased versions. This creates a larger group of responses.
  • Joint Group-Relative Normalization: Advantages are computed by pooling rewards across this entire expanded group. This means that even if the original question yields uniform rewards, the rephrased versions are likely to produce varied rewards, thus mitigating gradient vanishing.
  • Anchored Importance Ratios: Crucially, the model updates are still anchored to the original question. This ensures that the learning signal is directed towards solving the intended problem, while the diverse rephrasings encourage exploration of different reasoning paths, combating diversity collapse.

Datasets

  • Training Data: The primary dataset used for training is the MATH dataset [14], which contains 7498 high school-level mathematics questions.
  • Evaluation Benchmarks:
    • Competition-level mathematics: AMC [15], OlympiadBench [16], AIME24, and AIME25 [15].
  • Out-of-distribution evaluation: Minerva [17] and GPQA-Diamond [18].
  • Additional Data for Scaling Experiments: The OpenR1-MATH-220k dataset [37] was used to simulate training on larger datasets.

Key Insights and Ideas

  • Rephrasing as a "Difficulty Shifter": The core idea is that rephrasing a question, even while preserving its meaning, can subtly alter the model's perceived difficulty. This shift is key to generating varied rewards and thus a useful training signal.
  • Joint Normalization is Crucial: Pooling responses and normalizing rewards across the original and rephrased questions is more robust to gradient vanishing than normalizing per rephrasing. This is because it requires all (N+1) * G responses to have the same reward to fail, which is much less likely.
  • Diversity Collapse is a Persistent Problem: The paper highlights that standard GRPO's failure modes (gradient vanishing and diversity collapse) intensify during training, making the proposed solution increasingly important as training progresses.
  • Data Efficiency: TA-GRPO offers a way to achieve better performance without simply throwing more data at the problem, making it a more economical approach.

Key Results

  • Consistent Pass@k Improvements: TA-GRPO consistently improves pass@k scores across four different LLMs (Qwen3-1.7B, Qwen3-4B, Llama-3.2-1B, Llama-3.2-3B) and all six tested benchmarks.
  • Significant Gains: Average pass@32 gains over standard GRPO reached up to 5.09 points on competition math and 4.74 points on out-of-distribution benchmarks for the Qwen3 models. Peak gains were observed on specific benchmarks, e.g., 9.98 points on AIME24 for Qwen3-4B.
  • Data Efficiency: TA-GRPO with the original training data achieved pass@32 scores comparable to baselines trained on up to 2.5x more data.
  • Mechanism Validation: Experiments confirmed that rephrasing broadens the reward distribution and increases reasoning diversity, directly addressing the identified GRPO failure modes.

Limitations and Future Work

  • Reliance on GPT-4-Turbo: The method currently relies on GPT-4-Turbo for question rephrasing, which incurs API costs. Future work could explore using comparable open-weight models.
  • Training Time Overhead: TA-GRPO introduces additional overhead in training time due to generating and processing responses from rephrased questions.
  • Input Distribution as a Lever: The paper suggests that the input distribution (how questions are presented) is a third important axis for RL post-training, alongside the loss and reward axes. Combining TA-GRPO with other GRPO advancements is a natural next step.

Summary updated

Abstract

Large language models trained via next-token prediction are fundamentally pattern-matchers: sensitive to superficial phrasing variations even when the underlying problem is identical. Group Relative Policy Optimization (GRPO) was designed to improve reasoning, but in fact it worsens this situation through two failure modes: diversity collapse, where training amplifies a single solution strategy while ignoring alternatives of gradient signal, and gradient diminishing, where a large portion of questions yield zero gradients because all rollouts receive identical rewards. We propose TA-GRPO (Transform-Augmented GRPO), which generates semantically equivalent transformed variants of each question (via paraphrasing, variable renaming, and format changes) and computes advantages by pooling rewards across the entire group. This pooled computation ensures mixed rewards even when the original question is too easy or too hard, while training on diverse phrasings promotes multiple solution strategies. We provide theoretical justification showing that TA-GRPO reduces zero-gradient probability and improves generalization via reduced train-test distribution shift. Experiments on mathematical reasoning benchmarks show consistent Pass@k improvements, with gains up to 9.84 points on competition math (AMC12, AIME24) and 5.05 points on out-of-distribution scientific reasoning (GPQA-Diamond).
BibTeX
@misc{le2026transform,
  author = {Khiem Le and Youssef Mroueh and Phuc Nguyen and Chi-Heng Lin and Shangqian Gao and Ting Hua and Nitesh V. Chawla},
  title = {{Transform-Augmented GRPO Improves Pass@k}},
  year = {2026},
  eprint = {2601.22478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.22478},
}
Figure