This paper tackles a critical bottleneck in training modern reasoning models (like DeepSeek-R1 or OpenAI o1): the massive computational cost of Reinforcement Learning from Verifiable Rewards (RLVR). The authors introduce L2RR (Learning-to-Reweight Rollout), a method that makes training more efficient and effective by bridging the gap between training data and real-world target domains.
TL;DR
L2RR optimizes the training of reasoning models by using limited, privacy-preserving feedback from a target domain to prioritize the most useful training prompts. It significantly improves downstream performance on math and coding benchmarks while reducing the number of expensive model "rollouts" needed during training.
What Problem Does This Paper Address?
Current state-of-the-art reasoning models rely on Group Relative Policy Optimization (GRPO). In GRPO, the model generates a group of responses for a single prompt, and a verifier (like a math checker) scores them. This is expensive because generating these "rollouts" consumes the vast majority of training compute.
Existing methods try to save compute by skipping "easy" or "useless" prompts based on training performance. However, they suffer from two major flaws:
- Dataset Mismatch: Training data (source) rarely matches the actual distribution of the test data (target). A prompt that seems "useful" during training might be irrelevant to the target task.
- Privacy Constraints: In many real-world scenarios (like medical or legal applications), the trainer cannot access the target data directly due to privacy laws.
Methods
The authors propose a bi-level optimization framework called L2RR. Instead of sampling training prompts uniformly, L2RR learns a discrete sampling distribution ($q_t$) that evolves over time.
- Target-Guided Sampler: L2RR uses aggregated, directional derivatives from a small, held-out target dataset. This allows the trainer to understand which source prompts help the model generalize to the target domain without ever seeing the actual target prompts or responses.
- Rollout-Waste Penalty: It tracks "advantage collapse"—situations where all responses in a group get the same score (all right or all wrong). Since these provide zero gradient for learning, L2RR penalizes these prompts to save budget.
- Dynamic Rollout Allocation: Rather than using a fixed group size (e.g., always 8 responses), L2RR dynamically allocates more rollouts to prompts that have a high impact on target-domain alignment and fewer to those that are likely to be "wasteful."
Datasets
The researchers evaluated L2RR across multiple scales (1.5B to 7B parameters) using:
- Source Training Sets: DAPO+MATH and a 30k subset of OPEN-R1.
- Target Evaluation Benchmarks: Six major math reasoning sets including Math500, AIME24, AMC, Minerva Math, Gaokao, and Olympiad Bench.
- Cross-Domain Testing: They also tested the method on code generation using CodeContests as the source and HumanEval+/MBPP+ as the target.
Key Insights and Ideas
- Target Feedback as a Compass: Even limited, aggregated feedback is enough to steer the training process. By aligning training gradients with target-domain needs, the model learns features that generalize better.
- Efficiency via Reallocation: Efficiency isn't just about doing less; it's about doing more where it matters. By giving "high-influence" prompts a larger share of the rollout budget, the sampler gets a cleaner signal for its updates.
- The "Solved" Prompt Problem: Prompt difficulty is a moving target. As a model improves, prompts that were once useful become "solved" and start yielding zero-advantage rollouts. L2RR’s history-dependent counter effectively identifies and suppresses these "zombie" prompts.
Key Results
L2RR consistently outperformed strong baselines like GRESO and AR3PO across all models and datasets.
- Performance Boost: On the Qwen2.5-Math-1.5B model, L2RR improved average accuracy from 47.7% to 50.3% while using significantly fewer rollouts than the standard DAPO baseline.
- Rollout Efficiency: L2RR achieved a "Rollout Ratio" (RO ratio) of up to 3.5x, meaning it achieved better results while using a fraction of the total responses compared to standard methods.
- Scaling with Data: As shown in the table below, performance improves monotonically as the size of the held-out target set increases, though even a tiny 4% slice provides substantial gains.
The main performance comparison is summarized in Table 1: Table 1: Performance comparison across six benchmarks.
Limitations and Future Work
- Target Access: The method assumes a "target owner" exists who can provide aggregated feedback. In some decentralized or purely exploratory settings, this might not be available.
- Formal Privacy: While L2RR doesn't share raw prompts, the authors acknowledge that providing formal privacy guarantees (like Differential Privacy) for the gradient feedback is an important area for future research.
- Computational Overhead: While L2RR saves on rollout costs, the bi-level optimization adds some complexity to the training loop, though the authors found this to be negligible compared to the savings in generation.
Summary updated