Step-Level Truncated Importance Weights for Off-Policy Chain-of-Thought Training

Reinforcement learning on math problems now trains language models from responses that were generated by an older copy of the model. This happens because, in large training clusters, generation and training run asynchronously. To correct for the mismatch, the standard tool is importance sampling: each sample is reweighted by how much more or less likely the current model is to produce it than the old model was. The trouble is that a response is thousands of tokens long, and the per-token ratios either get multiplied together, which explodes or vanishes, or get clipped one token at a time, which ignores that the tokens form a single argument.

The literature disagrees about what to do. One paper says the correction must act on the whole-sequence weight. Another says the correction can be dropped almost entirely. We propose a middle granularity that neither side has tested: compute importance ratios over whole reasoning steps, meaning the sentences or lines of a chain-of-thought, and truncate those ratios at 1 in the style of classical multi-step off-policy learning. We pair this method with a direct measurement of how closely each estimator's gradient matches the true on-policy gradient. That measurement settles the dispute whichever estimator wins.

Core Claim

Chain-of-thought prompting, which has the model write intermediate reasoning before its answer, gives every response a natural segmentation into steps. We claim this segmentation is the right unit for off-policy correction, and the argument is mechanical.

Within a step, adjacent tokens are strongly correlated. Their per-token ratios scatter above and below 1 and largely cancel. Truncating each token's ratio separately, as token-level clipping does, discards the above-1 half of that scatter at every position. Over thousands of tokens this builds up a systematic downward bias. Truncating only after multiplying the ratios within a step lets the cancellation happen first. Meanwhile, the number of factors in the cross-step product drops from thousands of tokens to tens of steps. This also matches where reasoning actually goes wrong: one bad step derails the rest of the chain, as the error-compounding literature describes (Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?). Step boundaries are therefore where the old and new policies plausibly diverge.

Concretely, let a response to prompt $x$ consist of steps $s_1,\dots,s_K$. Let $\pi$ be the current policy and $\mu$ the stale policy that generated the response. The step ratio is $r_k = \pi(s_k \mid x, s_{<k}) / \mu(s_k \mid x, s_{<k})$, which is the product of that step's token ratios. The estimator weights each step's log-likelihood gradient by

$$w_k = \min(\bar c, r_k)\prod_{j<k}\min(1, r_j),$$

where $\bar c$ caps the current step's ratio. The product over earlier steps never exceeds 1, so it cannot explode. The update is $\sum_k w_k A_k \nabla \log \pi(s_k \mid x, s_{<k})$, where $A_k$ is an advantage, a measure of how much better than average the outcome was.

The prefix truncation follows Retrace($\lambda$) (Safe and Efficient Off-Policy Reinforcement Learning), which truncated per-step traces at 1 and proved safe convergence for Atari control. That work never touched language models. Our method lifts its traces from environment actions to reasoning steps.

Dropping the ratios of future steps introduces bias when the reward arrives only at the end. The standard remedy is a per-decision control variate (Per-decision Multi-step Temporal Difference Learning with Control Variates). Here, that means subtracting from the reward an estimate of the expected reward given the prefix up to step $k$. We treat this as a separately ablated component rather than a necessity.

Positioning Against Closest Work

Three papers define the tension this proposal tries to resolve.

VESPO argues that token-level clipping breaks sequence-level credit assignment. It reshapes the full sequence weight with a soft kernel, $\phi(W)=W^{c_1}\exp(c_2(1-W))$, and stays stable up to 64x staleness. It never looks inside the chain.

Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm finds the opposite. Its variants that omit importance sampling match GRPO, the group-relative policy gradient method used for reasoning models. It attributes stability to update clipping, which it widens to 0.6–2.0. It explicitly leaves step-level credit assignment open.

Ctrl-R steers rollouts toward behaviors such as backtracking and then corrects with a uniform power $\rho_t^\beta$ on every token. It finds $\beta=0.2$ best and $\beta=1$ unstable.

Each of these three applies one rule uniformly at either the token or the sequence level. None asks whether the unit of correction should be the reasoning step.

A fourth paper motivates the idea that tokens should not all be treated alike. OCEAN partitions tokens into entity and connective sets and weights them against different behavior models. Its partition is by token type and only for knowledge-graph facts. Ours is by position in the argument and applies to mathematics.

Finally, AGRO unifies on- and off-policy data through a consistency condition on regularized rewards. It is a competing route to using stale data, but it also operates on whole generations.

Research Program

The work has three parts.

The first part is a measurement instrument. Take a small model, Qwen2.5-1.5B as used in the GRPO-myths study, at a fixed checkpoint. Fix a buffer of responses generated by a checkpoint 1, 4, 16 or 64 updates older. Estimate the true on-policy policy gradient from a large number of fresh rollouts per prompt on a fixed set of MATH and GSM8K prompts. Then compute every candidate off-policy estimator on the stale buffer and report two quantities against that reference: cosine similarity and mean-squared error.

The candidates are token-level clipping, no importance sampling, the VESPO kernel, Ctrl-R-style power scaling, and our step-level weights. The reference gradient is expensive but feasible at this scale. It directly answers the question the dispute is really about: which estimator points where the on-policy gradient points, and with how much noise. We also log the effective sample size of each weighting scheme as a function of response length.

The second part is the training algorithm. We implement the step-level estimator inside a standard group-relative training loop. Steps are segmented at line breaks, with a minimum length so that tiny fragments merge into their neighbors. The default advantage is the group-relative one. As an ablation, we add a per-step control variate whose prefix values are estimated by branching a few extra rollouts at step boundaries. This costs rollouts, so we report it at matched total generation budget.

The third part is a stress test where off-policy mass is concentrated by design: Ctrl-R's guided rollouts. Here a handful of injected steps carry almost all of the divergence. A step-level scheme can treat those steps differently from the free-running steps around them. A uniform $\beta$ cannot.

Baselines

The naive reference point is the training setup most practitioners use: GRPO with token-level clipping at 0.2, run at the same staleness. Every method should beat this baseline at high staleness or the question is moot.

The strongest competitor on stability is VESPO. It was designed for exactly this regime, operates on the full sequence weight, and is reported to keep training curves stable at up to 64x staleness and to give "+2.3% over the best baseline" on Qwen3-30B-A3B-Base. The file gives no absolute AIME or MATH-500 accuracies for VESPO, so no reported result was found on Lacuna for VESPO's per-benchmark accuracy. We will reproduce it.

The strongest competitor on simplicity is REC-ONESIDE-NOIS from the GRPO-myths paper: no importance correction, clipping widened to the 0.6–2.0 range it recommends. No reported accuracy number was found on Lacuna for this baseline either.

AGRO is included as the consistency-based alternative. Its reported figure is a "+7% performance lift" over the reference model on MATH with Llama 3 8B, compared with "+4%" for standard RL.

In the guided regime, the baseline is Ctrl-R at $\beta=0.2$. Ctrl-R reports average math gains of "+1.27% to +1.38%" over the DAPO baseline on Qwen3 1.7B and 8B trained on DAPO-Math-17K.

All methods share the same base model, training data, rollout count, staleness schedule and evaluation decoding settings. Each method gets the same hyperparameter budget of six configurations over its own key constants: $\bar c$ for ours, $c_1, c_2$ for VESPO, clip range for NOIS, and $\beta$ for Ctrl-R. Each configuration runs on three seeds.

Beating a baseline has two parts. First, higher mean accuracy averaged over AIME 2024, AIME 2025 and MATH-500 at 16x and 64x staleness, with non-overlapping seed intervals. Second, no increase in the rate of collapsed runs.

Two ablations isolate the key ingredient. The first uses token-level truncation with the identical formula, so each token is a "step". The second, and more important, uses random segmentation with the same number and length distribution of segments as our semantic steps. If random chunks perform as well as reasoning steps, the gain comes from aggregating before truncating, not from the chain-of-thought structure. We would report it that way.

Why not just…?

Why not just drop importance sampling and widen the clip range? This is the most serious threat. The GRPO-myths paper reports that the no-correction variant matches GRPO on GSM8K, MATH and ToolACE. Its evidence, however, is about final training curves, not gradient accuracy, and it does not test staleness as extreme as VESPO's 64x. The gradient-fidelity measurement tells us directly whether ignoring the ratios leaves a gradient error that grows with staleness. The 64x training runs tell us whether that error matters for accuracy. If no-correction matches us at every staleness level, we will report that step-level correction is unnecessary, which is itself a useful resolution of the dispute.

Why not just use VESPO's sequence-level kernel? VESPO avoids explosion by softly suppressing whole sequences with extreme weights. When one early step diverges, the entire response is down-weighted, including later steps the current policy would still produce. Our estimator down-weights only the steps after the divergent one, and does so in proportion to the divergence. The prediction that would distinguish the two is higher effective sample size at matched gradient error for long responses. If VESPO shows equal fidelity and accuracy, the extra machinery is not justified.

Why not just tune Ctrl-R's power factor? Raising every token ratio to $\beta$ flattens weights uniformly. It cannot tell apart an injected backtracking step, whose ratio is tiny, from the ordinary steps around it, whose ratios are near 1. In the guided stress test, we sweep $\beta$ with the same tuning budget as our cap $\bar c$. If tuned $\beta$ matches step-level weighting, uniform flattening is enough when divergence is concentrated, and we will say so.

Why not just train on fresh data only? The VESPO and GRPO-myths papers both motivate off-policy reuse by the idle time of synchronous generation in large clusters. AGRO's reported +7% versus +4% suggests off-policy data adds learning signal beyond throughput. We include a synchronous on-policy run at equal wall-clock time as an upper reference for accuracy at zero staleness.

Proposed Experiments

The experiments run in order of cost.

The first is the gradient-fidelity study on Qwen2.5-1.5B with MATH and GSM8K prompts. It is cheap and decisive. If step-level weights do not track the true gradient better than the alternatives as staleness grows, the training experiments are unlikely to show a gain, and we stop there with a measurement paper.

The second is training. We use Llama-3.2-3B-Instruct and Qwen3-8B-Base, the models in VESPO's evaluation, trained on DAPO-Math-17K at staleness 1, 4, 16 and 64. We evaluate on AIME 2024, AIME 2025 and MATH-500, with GSM8K as a sanity check. We report mean accuracy over several samples per problem, collapse rate across seeds, and effective sample size.

The third is the guided-rollout comparison against Ctrl-R on Qwen3-1.7B.

The decisive checks are three. First, gradient fidelity at 64x staleness must order the estimators in our favor. Second, semantic segmentation must beat random segmentation of matched length. Third, step-level training must at least match both VESPO and the no-correction baseline at 64x staleness under equal tuning budgets.

If the first two checks fail, the hypothesis that reasoning steps are the right unit is false. That is a clean negative result about the structure of chain-of-thought off-policy error, and we would report it as such.