Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR

RLxF @ ICML 2026 · OpenReview

TL;DR

This paper introduces REFT, a simple yet effective method to improve Reinforcement Learning with Verifiable Rewards (RLVR) by diversifying the first token generated after a reasoning marker. By strategically sampling diverse initial tokens, REFT significantly boosts performance on math reasoning tasks without altering the core RLVR training process.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful technique for training language models to reason without needing labeled examples. Instead, it relies on generating multiple "rollouts" (reasoning traces) for a given prompt, which are then scored by a verifier. A key challenge in RLVR is ensuring sufficient diversity among these rollouts. If all rollouts are too similar, the verifier provides a weak learning signal. Existing methods often try to increase diversity by adjusting sampling temperature, modifying prefixes, or selecting rollouts. However, this paper argues that these approaches overlook a crucial, yet simple, opportunity: the very first token generated after the reasoning marker (e.g., <think>). This initial token, often a discourse opener, is surprisingly "low-load" in semantic content but "high-leverage" in its impact on the entire subsequent reasoning trace. The paper aims to show that by diversifying this specific, often overlooked, first token, significant improvements in RLVR performance can be achieved.

Methods

The core contribution is REFT (Rollout Exploration with First-Token Diversification). REFT is a minimal modification to the rollout sampling process within an existing RLVR framework. Instead of drawing the first token from the policy's standard, often sharply peaked distribution, REFT:

  1. Identifies a candidate set: It takes the policy's own top-N most probable first tokens after the reasoning marker.
  2. Stratifies sampling: It uniformly samples K of these top-N candidates.
  3. Allocates rollouts: It then distributes the total rollout budget (G) evenly among these K selected first tokens, assigning G/K continuations to each.
  4. Maintains standard continuation: Crucially, the rest of the reasoning trace (the continuation) is sampled using the unchanged policy, conditioned on the selected first token.

This means REFT doesn't change the verifier, the reward function, the advantage estimator, the RL objective, or the model architecture. It's purely a change in how the initial step of the rollout is constructed. This approach contrasts with methods like increasing temperature, which can introduce harmful noise into later, more critical reasoning steps.

Datasets

The paper uses several standard math reasoning datasets for training and evaluation:

  • GSM8K: A dataset of elementary math word problems.
  • BigMath: A collection of math reasoning problems.
  • BigMath-Easy: A subset of BigMath.
  • MATH-500, AIME 2024, AIME 2025, AMC 2023: Datasets used for evaluation, covering various math competition levels.

No new benchmarks were introduced; the focus is on improving performance on existing ones.

Key Insights and Ideas

  • The "low-load, high-leverage" first token: The paper's central insight is that the first token after a reasoning marker, often a simple discourse opener, has surprisingly little semantic impact on its own but profoundly influences the entire subsequent reasoning path.
  • Decoupling of probability and correctness: For this first token, the model's confidence (high probability for a few tokens) is weakly correlated with actual correctness. This means under-sampled, low-probability first tokens can still lead to correct solutions.
  • REFT as a targeted intervention: By focusing diversification on this specific, low-risk, high-impact position, REFT can broaden exploration without introducing noise into more sensitive parts of the reasoning process.
  • Training diversity vs. inference diversity: REFT increases semantic diversity during training, leading to better overall performance. However, at inference time, the policy trained with REFT might become more concentrated, suggesting that the training diversity helps the model learn more robust and efficient reasoning paths.

Key Results

  • Consistent performance gains: REFT consistently improves performance across four base models (0.5B to 7B parameters), two RLVR algorithms (GRPO and DAPO), and three difficulty regimes.
  • Improved Pass@k metrics: REFT significantly boosts Pass@1 (single-sample correctness), Pass@8, and Pass@64 (larger-budget coverage) on datasets like GSM8K and BigMath. For example, on GSM8K, REFT improved GRPO and DAPO by several percentage points in Pass@1 and Pass@8.
  • Effectiveness across scales: Gains are observed even on smaller models (0.5B parameters), indicating that REFT's approach is not contingent on large model capacity.
  • Reduced "all-wrong" groups: Analysis shows REFT leads to fewer rollout groups where all completions are incorrect, suggesting more effective exploration.
  • Mitigation of over-crediting: REFT prevents the standard GRPO/DAPO objective from over-sharpening the policy's preference for a single, often arbitrary, first token.
  • Runtime efficiency: REFT introduces negligible runtime overhead and can even lead to slightly faster inference due to improved accuracy and shorter completions.

Limitations and Future Work

  • Focus on structured reasoning: The current study is limited to verifier-based mathematical reasoning with explicit <think>-style reasoning traces.
  • Generalization to other tasks: The authors suggest future work should investigate how this first-token decoupling translates to:
  • Open-ended generation.
  • Multilingual reasoning.
  • Tool-use tasks.
  • Tasks where the first token carries more semantic content.
  • Broader Impact: While REFT improves efficiency, the authors acknowledge that stronger reasoning capabilities have dual-use risks (e.g., academic dishonesty, unsafe automation) and emphasize the need for responsible deployment with robust safety evaluations.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) trains reasoning models without labeled trajectories, relying on grouped rollouts to expose the policy to alternative reasoning paths and a verifier to score them. Rollout diversity has accordingly emerged as a central bottleneck in RLVR, with most existing methods broadening exploration through temperature, prefix, or rollout-selection adjustments. We identify a structurally distinguished but overlooked position for broadening this diversity: the first token after the reasoning marker. The policy's first-token distribution exhibits a sharply peaked yet correctness-decoupled phenomenon, and this first token position can broaden the regions a rollout group covers without altering the correctness signal. We introduce REFT (Rollout Exploration with First-Token Diversification), a light addition to the RLVR pipeline that samples first tokens uniformly from the policy's own top-$N$ candidates and allocates rollouts evenly, leaving every other component unchanged. Trained on the resulting diversified rollouts, REFT improves aggregate Pass@1, Pass@8, and Pass@64 over DAPO and GRPO baselines across four base models (0.5B-7B) and three difficulty regimes.
BibTeX
@inproceedings{2026where,
  title = {{Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=hgMGfmmQ6u},
}
Figure