TL;DR
This paper introduces REFT, a simple yet effective method to improve Reinforcement Learning with Verifiable Rewards (RLVR) by diversifying the first token generated after a reasoning marker. By strategically sampling diverse initial tokens, REFT significantly boosts performance on math reasoning tasks without altering the core RLVR training process.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful technique for training language models to reason without needing labeled examples. Instead, it relies on generating multiple "rollouts" (reasoning traces) for a given prompt, which are then scored by a verifier. A key challenge in RLVR is ensuring sufficient diversity among these rollouts. If all rollouts are too similar, the verifier provides a weak learning signal. Existing methods often try to increase diversity by adjusting sampling temperature, modifying prefixes, or selecting rollouts. However, this paper argues that these approaches overlook a crucial, yet simple, opportunity: the very first token generated after the reasoning marker (e.g., <think>). This initial token, often a discourse opener, is surprisingly "low-load" in semantic content but "high-leverage" in its impact on the entire subsequent reasoning trace. The paper aims to show that by diversifying this specific, often overlooked, first token, significant improvements in RLVR performance can be achieved.
Methods
The core contribution is REFT (Rollout Exploration with First-Token Diversification). REFT is a minimal modification to the rollout sampling process within an existing RLVR framework. Instead of drawing the first token from the policy's standard, often sharply peaked distribution, REFT:
- Identifies a candidate set: It takes the policy's own top-N most probable first tokens after the reasoning marker.
- Stratifies sampling: It uniformly samples K of these top-N candidates.
- Allocates rollouts: It then distributes the total rollout budget (G) evenly among these K selected first tokens, assigning G/K continuations to each.
- Maintains standard continuation: Crucially, the rest of the reasoning trace (the continuation) is sampled using the unchanged policy, conditioned on the selected first token.
This means REFT doesn't change the verifier, the reward function, the advantage estimator, the RL objective, or the model architecture. It's purely a change in how the initial step of the rollout is constructed. This approach contrasts with methods like increasing temperature, which can introduce harmful noise into later, more critical reasoning steps.
Datasets
The paper uses several standard math reasoning datasets for training and evaluation:
- GSM8K: A dataset of elementary math word problems.
- BigMath: A collection of math reasoning problems.
- BigMath-Easy: A subset of BigMath.
- MATH-500, AIME 2024, AIME 2025, AMC 2023: Datasets used for evaluation, covering various math competition levels.
No new benchmarks were introduced; the focus is on improving performance on existing ones.
Key Insights and Ideas
- The "low-load, high-leverage" first token: The paper's central insight is that the first token after a reasoning marker, often a simple discourse opener, has surprisingly little semantic impact on its own but profoundly influences the entire subsequent reasoning path.
- Decoupling of probability and correctness: For this first token, the model's confidence (high probability for a few tokens) is weakly correlated with actual correctness. This means under-sampled, low-probability first tokens can still lead to correct solutions.
- REFT as a targeted intervention: By focusing diversification on this specific, low-risk, high-impact position, REFT can broaden exploration without introducing noise into more sensitive parts of the reasoning process.
- Training diversity vs. inference diversity: REFT increases semantic diversity during training, leading to better overall performance. However, at inference time, the policy trained with REFT might become more concentrated, suggesting that the training diversity helps the model learn more robust and efficient reasoning paths.
Key Results
- Consistent performance gains: REFT consistently improves performance across four base models (0.5B to 7B parameters), two RLVR algorithms (GRPO and DAPO), and three difficulty regimes.
- Improved Pass@k metrics: REFT significantly boosts Pass@1 (single-sample correctness), Pass@8, and Pass@64 (larger-budget coverage) on datasets like GSM8K and BigMath. For example, on GSM8K, REFT improved GRPO and DAPO by several percentage points in Pass@1 and Pass@8.
- Effectiveness across scales: Gains are observed even on smaller models (0.5B parameters), indicating that REFT's approach is not contingent on large model capacity.
- Reduced "all-wrong" groups: Analysis shows REFT leads to fewer rollout groups where all completions are incorrect, suggesting more effective exploration.
- Mitigation of over-crediting: REFT prevents the standard GRPO/DAPO objective from over-sharpening the policy's preference for a single, often arbitrary, first token.
- Runtime efficiency: REFT introduces negligible runtime overhead and can even lead to slightly faster inference due to improved accuracy and shorter completions.
Limitations and Future Work
- Focus on structured reasoning: The current study is limited to verifier-based mathematical reasoning with explicit
<think>-style reasoning traces. - Generalization to other tasks: The authors suggest future work should investigate how this first-token decoupling translates to:
- Open-ended generation.
- Multilingual reasoning.
- Tool-use tasks.
- Tasks where the first token carries more semantic content.
- Broader Impact: While REFT improves efficiency, the authors acknowledge that stronger reasoning capabilities have dual-use risks (e.g., academic dishonesty, unsafe automation) and emphasize the need for responsible deployment with robust safety evaluations.
Summary updated