Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards

Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi SugiyamaarXiv 2026 · arXiv

This paper tackles a critical bottleneck in training modern reasoning models (like DeepSeek-R1 or OpenAI o1): the massive computational cost of Reinforcement Learning from Verifiable Rewards (RLVR). The authors introduce L2RR (Learning-to-Reweight Rollout), a method that makes training more efficient and effective by bridging the gap between training data and real-world target domains.

TL;DR

L2RR optimizes the training of reasoning models by using limited, privacy-preserving feedback from a target domain to prioritize the most useful training prompts. It significantly improves downstream performance on math and coding benchmarks while reducing the number of expensive model "rollouts" needed during training.

What Problem Does This Paper Address?

Current state-of-the-art reasoning models rely on Group Relative Policy Optimization (GRPO). In GRPO, the model generates a group of responses for a single prompt, and a verifier (like a math checker) scores them. This is expensive because generating these "rollouts" consumes the vast majority of training compute.

Existing methods try to save compute by skipping "easy" or "useless" prompts based on training performance. However, they suffer from two major flaws:

  1. Dataset Mismatch: Training data (source) rarely matches the actual distribution of the test data (target). A prompt that seems "useful" during training might be irrelevant to the target task.
  2. Privacy Constraints: In many real-world scenarios (like medical or legal applications), the trainer cannot access the target data directly due to privacy laws.

Methods

The authors propose a bi-level optimization framework called L2RR. Instead of sampling training prompts uniformly, L2RR learns a discrete sampling distribution ($q_t$) that evolves over time.

  • Target-Guided Sampler: L2RR uses aggregated, directional derivatives from a small, held-out target dataset. This allows the trainer to understand which source prompts help the model generalize to the target domain without ever seeing the actual target prompts or responses.
  • Rollout-Waste Penalty: It tracks "advantage collapse"—situations where all responses in a group get the same score (all right or all wrong). Since these provide zero gradient for learning, L2RR penalizes these prompts to save budget.
  • Dynamic Rollout Allocation: Rather than using a fixed group size (e.g., always 8 responses), L2RR dynamically allocates more rollouts to prompts that have a high impact on target-domain alignment and fewer to those that are likely to be "wasteful."

Datasets

The researchers evaluated L2RR across multiple scales (1.5B to 7B parameters) using:

  • Source Training Sets: DAPO+MATH and a 30k subset of OPEN-R1.
  • Target Evaluation Benchmarks: Six major math reasoning sets including Math500, AIME24, AMC, Minerva Math, Gaokao, and Olympiad Bench.
  • Cross-Domain Testing: They also tested the method on code generation using CodeContests as the source and HumanEval+/MBPP+ as the target.

Key Insights and Ideas

  • Target Feedback as a Compass: Even limited, aggregated feedback is enough to steer the training process. By aligning training gradients with target-domain needs, the model learns features that generalize better.
  • Efficiency via Reallocation: Efficiency isn't just about doing less; it's about doing more where it matters. By giving "high-influence" prompts a larger share of the rollout budget, the sampler gets a cleaner signal for its updates.
  • The "Solved" Prompt Problem: Prompt difficulty is a moving target. As a model improves, prompts that were once useful become "solved" and start yielding zero-advantage rollouts. L2RR’s history-dependent counter effectively identifies and suppresses these "zombie" prompts.

Key Results

L2RR consistently outperformed strong baselines like GRESO and AR3PO across all models and datasets.

  • Performance Boost: On the Qwen2.5-Math-1.5B model, L2RR improved average accuracy from 47.7% to 50.3% while using significantly fewer rollouts than the standard DAPO baseline.
  • Rollout Efficiency: L2RR achieved a "Rollout Ratio" (RO ratio) of up to 3.5x, meaning it achieved better results while using a fraction of the total responses compared to standard methods.
  • Scaling with Data: As shown in the table below, performance improves monotonically as the size of the held-out target set increases, though even a tiny 4% slice provides substantial gains.
Table 3: Effect of held-out target set size for sampler learning. Accuracy (%) on six reasoning benchmarks for Qwen2.5-Math-1.5B trained on DAPO + MATH. We vary the size of held-out target dataset to update qt.

The main performance comparison is summarized in Table 1: Table 1: Performance comparison across six benchmarks.

Table 1: Performance (%) comparison across six math reasoning benchmarks. We train three models on source datasets DAPO+MATH (DM) and the Open R1 subset (OR1) and test on six benchmarks. # Rollout denotes cumulative source responses, in millions. RO ratio is the DAPO count divided by the method count. Accuracies are mean ± standard deviation over five training seeds.

Limitations and Future Work

  • Target Access: The method assumes a "target owner" exists who can provide aggregated feedback. In some decentralized or purely exploratory settings, this might not be available.
  • Formal Privacy: While L2RR doesn't share raw prompts, the authors acknowledge that providing formal privacy guarantees (like Differential Privacy) for the gradient feedback is an important area for future research.
  • Computational Overhead: While L2RR saves on rollout costs, the bi-level optimization adds some complexity to the training loop, though the authors found this to be negligible compared to the savings in generation.

Summary updated

Abstract

Recently, reinforcement learning from verifiable rewards (RLVR) is practically very important, where the leading approach is group relative policy optimization (GRPO) that typically faces a trade-off between rollout efficiency and downstream performance. To improve efficiency while preserving performance, existing selective rollout methods mainly exploit only source-side training dynamics to optimize the discrete distribution over source prompts. However, in practice, the source training dataset is rarely identically distributed to the target test dataset. When such a mismatch exists, due to legal or privacy constraints, it is usually not acceptable to share target prompts themselves with the RLVR trainer. Nevertheless, it is still possible for the target data owner to provide limited aggregated feedback from a small held-out target set without revealing target prompts or rollout traces. In this paper, we propose learning-to-reweight rollout (L2RR) that explicitly incorporates target-side performance feedback into GRPO. Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bi-level objective that maximizes accuracy on a held-out target dataset, while simultaneously penalizing rollouts that provide little incremental learning information. Empirical studies on six math reasoning benchmarks and three model scales show that L2RR improves target-domain performance and increases rollout efficiency.
BibTeX
@misc{zhang2026dataset,
  author = {Zhen-Yu Zhang and Jiandong Zhang and Huaxiu Yao and Gang Niu and Masashi Sugiyama},
  title = {{Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards}},
  year = {2026},
  eprint = {2611.00586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00586},
}
Figure