Nominal-Prefix Rollouts: Keeping Reasoning RL Informative by Choosing Where Rollouts Start

When a language model is trained with reinforcement learning on math problems, most of the compute goes into generating answers token by token. Much of that is wasted. A common method, Group Relative Policy Optimization (GRPO), learns only from differences between several attempts at the same problem. If all eight attempts fail, or all eight succeed, the update is zero. On hard problems the model fails every time and learns nothing. On easy problems it succeeds every time and also learns nothing. Current fixes skip these problems before generating anything. That saves compute, but it permanently throws away the hardest problems, which are the ones the model most needs to learn from.

We propose a different fix, adapted from a standard idea in control engineering. In two-degree-of-freedom control, one part of the controller follows a planned reference path and a separate feedback part corrects deviations from it. Applied here, each training problem gets a nominal trajectory: a known-good solution, either a correct answer the model produced earlier or the reference solution shipped with the dataset. The model then generates only the continuation from a chosen point partway along that solution. A small controller moves that starting point earlier or later for each problem so that roughly half the attempts succeed, which is where the learning signal is strongest. As the model improves, the controller pushes the starting point back toward the beginning of the problem, until the model solves it from scratch. The bet is that the model gets nonzero learning signal on problems that would otherwise be discarded, while generating far fewer tokens, because a prefix taken from a known solution is processed in one parallel pass rather than generated one token at a time.

Core Claim

Wasted rollouts are not only a problem to filter out. They can be turned into useful signal by controlling the state each rollout starts from, rather than only choosing which prompts to roll out. The specific claim is that a per-prompt controller can hold group success rates near one half throughout training. This controller has two parts: a feedforward prediction of the starting depth, taken from the prompt's history, and a feedback correction from the success rate it just observed. Holding success near one half should reach the same final from-scratch accuracy as GRPO with fewer generated tokens than prompt filtering. It should also make headway on problems the base model never solves.

The target follows from the shape of the reward. With binary rewards and a group success probability $p$, the per-group reward variance is $p(1-p)$. That variance scales GRPO's advantage signal and vanishes at $p \in \{0,1\}$, so keeping $p$ near $p^\ast = 0.5$ maximizes it. Let $d_i \in [0,1]$ be the fraction of the nominal solution for prompt $i$ that is handed to the model as a prefix. The depth is updated as

$$d_i^{(t+1)} = \mathrm{clip}\Big(\hat d_i^{\,\mathrm{ff}} + \delta_i^{(t)} + K\big(\hat p_i^{(t)} - p^\ast\big)\cdot(-1) ,\ 0,\ 1\Big) - \lambda t,$$

The terms are defined as follows:

  • $\hat d_i^{\,\mathrm{ff}}$ is a feedforward depth predicted from the prompt's MATH difficulty level and its past pass rates.
  • $\delta_i^{(t)}$ is the accumulated feedback correction.
  • $\hat p_i^{(t)}$ is the observed group success rate at the last visit to the prompt.
  • $K$ is a feedback gain. When success is too low, depth increases and the model is handed more of the solution. When success is too high, depth decreases.
  • $\lambda$ is a slow global decay that forces all depths toward zero, so that by the end of training every rollout starts from scratch.

The split mirrors the control literature. The feedforward term handles the predictable part of difficulty. The feedback term rejects the unpredictable part: drift in the policy itself as it trains.

Where the Idea Comes From and How It Differs

Three lines of control work supply the structure.

Robust Sampling Based Model Predictive Control with Sparse Objective Information introduces Tube-MPPI, a planner for sparse yes/no costs. It uses a rule called Nominal State Selection: when a disturbance pushes the system into a region where every option looks equally bad, the planner refuses to replan from that state. Instead it keeps the safe nominal plan and lets a feedback controller pull the system back. An all-zero rollout group is exactly such a flat region. Our response is the same: start from the nominal path rather than from the state where the model is lost.

Decoupled Data-Based Approach for Learning to Control Nonlinear Dynamical Systems (D2C) does two things. It optimizes a single nominal trajectory, then learns only local feedback around it. This cut training on a 16-dimensional swimmer from over 80,000 seconds with model-free reinforcement learning to about 3,600 seconds. D2C also constrains our design: it becomes fragile under large disturbances, where learning only near the nominal path misses what end-to-end RL finds. That is why we decay the depth to zero and do not stop at local learning.

Stochastic Online Optimization for Cyber-Physical and Robotic Systems supplies the right theoretical worry. Approximate gradients still converge only if they stay aligned with the true gradient, which that paper formalizes as a modeling-error modulus $\kappa < 1$. For us, gradients computed from prefix-conditioned states are approximations of the from-scratch gradient. We will measure their alignment directly rather than assume it.

On the reinforcement-learning-efficiency side, the closest work attacks the same waste in different ways:

  • Act Only When It Pays (GRESO) reports that the share of useful prompts falls to about 20% late in training. It skips predicted zero-variance prompts before generation. This is the motivating evidence for our work, but GRESO only removes problems and never makes a hard one learnable.
  • Dynamic One-Shot Policy Refinement (DoPR) probes problems with a single rollout and spends a full group only on the most uncertain one. It also selects prompts and does not reshape them.
  • SPEC-RL reuses rollouts from earlier epochs as speculative drafts, keeping the verified prefix and regenerating after the first rejection. It is the nearest mechanism, because it also continues from old prefixes. However, its cut point is decided by whether the current policy agrees with the old tokens, governed by a fixed lenience scalar. The goal is to reproduce the current policy's output faster, not to place the start where learning signal is maximal. SPEC-RL also cannot help in the first epoch, since its cache is empty. Our use of dataset reference solutions removes that cold-start gap.
  • Systems work such as ReSpec and EfficientRollout speeds up each generated token. It is complementary and could be stacked on top of our method.

Research Program

We will build the method on a standard GRPO training loop for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, trained on the 7,500-problem training split of the MATH dataset. MATH ships step-by-step reference solutions, which serve as the nominal trajectory when the model has no correct answer of its own. When the model has produced a correct solution for a prompt, we prefer it. It is closer to the model's own style, so the prefix is less out-of-distribution.

Nominal solutions are cut only at step boundaries (line breaks or sentence ends), so the model never resumes mid-formula. Each prefix is fed through the model in one parallel pass, which is cheap in the same way teacher forcing is cheap. Only the suffix is sampled. The policy gradient is applied only to the generated suffix tokens. A fixed fraction of rollouts, starting at one quarter, always starts from depth zero, so the training signal never entirely loses contact with the true starting distribution.

Three analyses accompany the method. First, a gradient-alignment probe: on a fixed set of held-out prompts, we compare the GRPO gradient from prefix-conditioned rollouts with the gradient from a large from-scratch batch, using cosine similarity, at regular intervals during training. This is the empirical stand-in for the $\kappa < 1$ condition. Second, a careful compute ledger. We count generated tokens separately from prefilled tokens and also report wall-clock time, because the method's advantage rests on prefill being far cheaper than generation. Third, a per-prompt trajectory analysis. For prompts that were all-zero under the base model, we track how depth falls over training and whether they eventually get solved from depth zero.

Baselines

The naive reference point is plain GRPO with eight rollouts per prompt, no filtering and no prefixes. It anchors both accuracy and cost. On Qwen2.5-Math-1.5B, DoPR reports a standard GRPO score of 71.8.

The strongest efficiency competitor on exactly this question is GRESO, because it attacks the same zero-variance waste with a cheap pre-generation filter. On Qwen2.5-Math-7B it reports a 2.4x rollout speedup, a 2.0x total training speedup, and 57.5% average accuracy against a 57.8% baseline. We will rerun it on our data, with the same models, the same total generated-token budget, and the same hyperparameter search budget for both methods. Beating it means two things. We must match or exceed its average accuracy across MATH-500, AIME24, AMC, OlympiadBench and Minerva Math at equal or lower generated-token count. We must also show a clear gain on the subset of training-distribution problems the base model never solves, where GRESO by construction learns nothing.

DoPR is the strongest accuracy-per-sample competitor, reporting 73.5 on Qwen2.5-Math-1.5B. We will run it under the same token budget. Beating it means higher accuracy at matched generated tokens.

SPEC-RL is the closest mechanism and is run with its published lenience setting. It reports a 2.31x average rollout speedup. No reported result was found on Lacuna for SPEC-RL on MATH training with Qwen2.5-Math-1.5B specifically, so we will produce one. Because SPEC-RL is largely orthogonal to ours, we also run the two combined.

Two ablations isolate the key ingredients. The first uses a fixed depth schedule that decays uniformly with no per-prompt feedback; this tests whether the controller matters at all. The second keeps the controller but uses only dataset reference solutions, never the model's own successes; this tests how much prefix style mismatch costs.

Why not just…?

Why not just skip the uninformative prompts, as GRESO does? Skipping is cheaper per step, and if all the useful learning happens on medium-difficulty problems, skipping may suffice. That is a real possibility. GRESO's own near-parity accuracy shows that skipping does not hurt the benchmarks it measured. The case against it rests on the hard tail. ProRL reports tasks where the base model has a 0% success rate and only prolonged RL, costing about 16,000 H100 hours, reaches them. Skipping can never reach them. The experiment that settles this is the never-solved subset. If our method shows no gain there over GRESO at matched tokens, the idea fails.

Why not just fine-tune on the reference solutions with teacher forcing? This is the most serious threat. It uses the same nominal solutions and is fully parallel, and Teacher Forcing Recovers Reward Functions for Text Generation argues that supervised training already carries reward-like signal. However, the MATH paper found that supervised training on reasoning scales poorly. The RL4LMs study found that reinforcement learning beat supervised models trained on five times more data. We will run supervised fine-tuning on the same solutions as a starting point, followed by standard GRPO, with matched total compute. If this pipeline equals our method, the controller adds nothing and we will say so.

Why not just reuse old rollouts with speculative verification, as in SPEC-RL? SPEC-RL makes rollouts cheaper but does not change which rollouts carry signal. A reused prefix of an all-failed group still yields an all-failed group. The combined run tests whether the two savings add. If SPEC-RL alone already yields the same effective-prompt ratio as our method, our contribution reduces to its cold-start fix.

Proposed Experiments

The main experiment trains each method to a fixed generated-token budget, set to the GRPO baseline's full run. We report accuracy on the five held-out benchmarks against generated tokens and against wall-clock time. We also report pass@k at large k, to check whether the method widens the set of solvable problems rather than only sharpening the answers the model already finds.

Three decisive checks follow:

  • Effective-prompt ratio. The fraction of groups with nonzero reward variance should stay well above the roughly 20% late-training level GRESO reports. If the controller cannot hold it there, the mechanism is not working.
  • Gradient alignment. The cosine similarity between prefix-conditioned and from-scratch gradients must stay positive and rise as depth decays. If it falls to zero or below, the prefix states are teaching the wrong thing, which is the failure mode the $\kappa$ analysis predicts.
  • Prefix dependence. Final from-scratch accuracy must match that of a model evaluated with prefixes. A large gap would mean the policy learned to lean on prefixes it will not have at test time, the language-model analog of D2C's fragility away from its nominal path.

A secondary run on GSM8K, which also ships worked solutions, tests whether the controller gains carry over to shorter problems, where prefixes save less.