Training a language model to debug its own code with reinforcement learning (RL, learning from reward by trial and error) is expensive. Each training episode alternates between the model writing a program and a sandbox running it against tests, and a single episode may contain four or five such rounds. To keep GPUs busy, large systems run these episodes asynchronously. Many workers generate episodes while a central learner keeps updating the model. The data the learner receives therefore comes from older versions of the model, a mismatch called policy lag. In multi-turn debugging the lag is not even uniform. Turn 1 of an episode may come from a model eight updates old and turn 3 from one a single update old. Slow-running programs, which are often the interesting algorithmic cases, spend longer in the sandbox and arrive staler than quick syntax failures. Existing corrections for lag either throw this data away or reweight whole sequences with importance ratios, which blow up over long multi-turn horizons.
This proposal rests on a simple observation. The expensive and informative part of a stale debugging episode is the buggy program together with its real execution feedback, and that feedback stays true no matter which model version wrote the program. Only the model's choices go stale; the situations it created do not. We propose to treat every stale turn prefix (problem, earlier code, compiler or test output) as a reusable starting state. From that state we draw a fresh repair from the current model, execute it once, and train on it with an ordinary near-on-policy update. Old sandbox executions are reused, fresh actions carry no importance-ratio correction, and slow algorithmic states are kept rather than filtered out.
Core Claim
In asynchronous multi-turn code RL, splitting stale episodes into "state" and "action" and regenerating only the final repair turn should reach the accuracy of synchronous multi-turn GRPO with substantially fewer sandbox executions and less wall-clock time. GRPO (group relative policy optimization) is the reward-normalized policy-gradient method used by most recent code RL work. The split should also remain stable at lags where token-level or turn-level importance sampling degrades. The claim is specific enough to fail in three ways. The regenerated turns may be too narrow a signal to teach first-draft quality. States produced by old policies may drift away from what the current model actually visits. Or the whole gain may be explained by one-step imitation learning on the same state bank, with RL adding nothing.
Relation to Prior Work
Three papers motivate the gap by showing that stale data is valuable but hard to use. Prosperity before Collapse shows that stale rollouts up to 256 updates old still carry strong signal once second-moment masking replaces PPO clipping. D-ARL finds that only 18.1% of high-quality samples come from the newest policy for Qwen3-1.7B, and selects replay samples to reduce estimator variance. Gradient-Aware Scheduling argues that importance-sampling variance grows exponentially with task difficulty under staleness and tightens lag budgets on hard tasks. All three treat an episode as a single-turn sequence with one staleness scalar and still correct or discard the actions in it. We never correct stale actions at all. We keep their states and replace their actions.
Two papers constrain the design. ST-PPO shows that token-level importance sampling in multi-turn agents over-optimizes formatting tokens and collapses, and proposes turn-level ratios. Our approach sidesteps the ratio product entirely, but ST-PPO's turn-level correction is the natural competing fix and serves as a baseline. The same work warns that off-policy states can mislead training. That is why we add a relevance filter on prefixes, described below.
Three papers supply the method we build on, and all of them run synchronously. MURPHY builds feedback-conditioned rollout trees and propagates the best descendant reward back to earlier drafts. We reuse its training data (1,000 KodCode problems), its benchmarks, and its credit rule. TGPR moves tree search into training and reports a 37.5% wall-clock overhead for doing so. That overhead is exactly what asynchronous state reuse targets. Multi-Turn Code Generation Through Single-Step Rewards, μCODE, argues that code repair is "one-step recoverable" and trains single-step corrections by imitation against a learned verifier. Our state-as-context view borrows this one-step framing. The difference is that we keep a policy-gradient update on fresh actions and apply it inside an asynchronous pipeline, and whether that difference matters is tested directly.
Two papers bound the approach from outside. VERINA shows refinement gains plateau and can thrash over many rounds. This suggests states deep in long, failing chains are low value, so our state bank weights turn depth accordingly. TideRL solves the systems side of asynchronous multi-turn agents, including cache preservation and readiness-aware scheduling, but uses unmodified RL objectives. We take its style of pipeline as the substrate and change the objective.
Research Program
The system has three parts. The first is an asynchronous rollout pool that runs full multi-turn repair episodes, as MURPHY does. Every turn prefix in these episodes, with its cached execution output, test results, and the model version that produced each turn, goes into a state bank. Execution time is recorded as well, so we can measure whether slow programs are over- or under-represented.
The second part is a refresh sampler. It draws states from the bank and has the current model produce a group of $G$ candidate next repairs per state. Each candidate is executed once and scored by test pass rate. A GRPO update is then applied to only those new tokens. The lag on these actions is just the delay between the refresh and the gradient step, which we keep to a few updates, so standard clipping suffices. Episodes whose first turn is fresh are still trained with MURPHY's full tree credit, so first-draft generation keeps receiving signal.
The third part is a prefix relevance filter. It addresses the fact that states generated by an old model may be ones the current model would never reach, such as syntax errors the model has since stopped making. For a prefix $h$ with model-written tokens $x_1,\dots,x_n$, we compute
$$M(h)=\frac{1}{n}\sum_{i=1}^{n}\left(\log \pi_\theta(x_i\mid x_{<i}) - \log \mu(x_i\mid x_{<i})\right)^2,$$
where $\pi_\theta$ is the current model and $\mu$ is the model version that wrote each token. This is M2PO's second-moment statistic applied to states rather than to actions. States with $M(h)$ above a threshold are sampled less often. The weight is used only as a sampling probability, never multiplied into gradients, so it cannot cause gradient variance to explode.
Alongside the method, we will measure how turn-varying lag arises in practice. This includes the distribution of per-turn staleness within episodes, its correlation with sandbox execution time, and how often the fast-failure bias identified in the research file actually appears in the batches each method trains on.
Baselines
The naive reference point is the base instruction-tuned model, a 7B Qwen-family coder so that TGPR's Qwen-7B results give a scale anchor. It is evaluated with the same multi-turn self-repair protocol (up to four feedback rounds) and no RL.
The strongest existing method on this question is synchronous MURPHY. It is the only method in the file that trains multi-turn repair with tree credit on these benchmarks. It reports up to an 8% absolute pass@1 gain over compute-matched single-turn GRPO on HumanEval, MBPP and BigCodeBench-Hard MURPHY. Per-benchmark MURPHY numbers are not given in the file, so we will reproduce it with identical data (1,000 KodCode problems), model, group size, and hyperparameter search budget. Beating it means matching its final pass@1 within noise across three seeds while using at least 30% fewer sandbox executions and less wall-clock, or exceeding it at equal wall-clock. The 30% bar is set near D-ARL's reported step savings, 65.3% of baseline steps, which is the file's closest analogue for what asynchronous reuse buys D-ARL.
The strongest asynchronous competitors are MURPHY's trees run through the same asynchronous pool, with stale turns corrected in one of two ways. One is M2PO's second-moment masking applied token-wise Prosperity before Collapse. The other is ST-PPO's turn-level importance ratios ST-PPO. A third variant adds GAS-style difficulty-dependent staleness rejection. For single-turn code, GAS reports 60.1% combined Pass@1 against 31.8% for standard asynchronous RL Gradient-Aware Scheduling, though that is a different, single-turn setting and not directly comparable. All asynchronous arms share the same pool, lag distribution, and sandbox budget.
Two ablations isolate the key ingredient. The first removes the prefix relevance filter and samples the bank uniformly. The second replaces the refresh-and-GRPO step with μCODE-style imitation of the best fresh repair per state, holding the bank and execution count fixed.
Why not just…?
Why not just run synchronous MURPHY and accept the cost? Synchronous multi-turn training leaves hardware idle while sandboxes run. TideRL reports up to 77.6% idle waiting time eliminated in multi-turn agent workloads when moving away from synchronous scheduling TideRL. TGPR reports a 37.5% training-time overhead just from tree rollouts TGPR. If our method only matches MURPHY's accuracy at equal wall-clock, this alternative wins, and the headline comparison is designed to show that outcome if it holds.
Why not just apply turn-level importance sampling or M2PO to the full stale trees? This is the most direct competitor and could work at modest lag. Two points in the file cut against it. GAS argues that correction variance grows exponentially with difficulty, and repair turns on hard bugs are exactly the hard case. Importance ratios also multiply across turns whose behavior policies differ, which is the regime where ST-PPO saw collapse without its fix. Our lag sweep settles the question. If turn-level importance sampling stays level with state reuse even at large lag, the state/action split adds little and we will report that.
Why not just use μCODE, since treating turns as one-step states is its idea? This is a real threat, because μCODE already exploits one-step recoverability and avoids policy lag by avoiding RL. Its reported gains come in a synchronous imitation setting with Best-of-N inference: 16.3% on CodeContests, and +5.8% over rejection fine-tuning on HumanEval with a 1B model Multi-Turn Code Generation Through Single-Step Rewards. TGPR and MURPHY argue that greedy one-step fixes stagnate on hard bugs. The imitation ablation on an identical state bank is the decisive test. If imitation matches GRPO refresh, the contribution shrinks to "asynchronous state banking helps μCODE-style training," and the note will say so.
Why not just discard stale data with a hard cutoff? A cutoff throws away the slowest and often most algorithmic episodes, since those are the ones that go stale. This is the selection bias the file identifies, and D-ARL's 18.1% figure shows fresh data is not where most good samples are. We will run a cutoff at lag one as a baseline and report the execution-time distribution of the programs it keeps.
Proposed Experiments
Training uses MURPHY's 1,000 KodCode problems. Evaluation covers HumanEval+, MBPP+, BigCodeBench-Hard, and LiveCodeBench in both code-generation and self-repair modes, with pass@1 under greedy single-pass and four-round repair. As external context, TGPR's Qwen-7B reports 41.2 on LiveCodeBench and 77.6 on MBPP TGPR. LiveCodeBench's self-repair mode lifted GPT-4-Turbo from 24.5% to 36.9% LiveCodeBench, which shows the repair axis is measurable there.
The main comparison plots pass@1 against three cost axes: sandbox executions, generated tokens, and wall-clock. The comparison runs across all baselines with three seeds.
A lag sweep then controls maximum rollout lag at 1, 8, 32, and 128 updates. We expect the importance-sampling arms to degrade with lag while state reuse stays flat until the relevance filter starts rejecting most of the bank. The point where that rejection sets in tells us how quickly old states become irrelevant.
A first-draft check measures single-attempt pass@1 without repair. This tests whether training mostly on refreshed later turns starves the model's initial-generation skill relative to MURPHY.
A bias audit compares the execution-time and bug-type distributions of the states each method actually trains on. Its purpose is to check whether state reuse keeps slow algorithmic cases that the cutoff and importance-sampling arms under-weight.
The proposal fails if any of the following holds. Synchronous MURPHY dominates at equal wall-clock. The imitation ablation matches GRPO refresh. Or first-draft accuracy drops noticeably below MURPHY's.
References
- Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
- D-ARL: A Distribution-Matched Asynchronous Reinforcement Learning Framework for Language Reasoning
- Gradient-Aware Scheduling: Coupling Curriculum and Staleness for Async Reinforcement Learning
- ST-PPO: Stabilized Off-Policy Proximal Policy Optimization for Multi-Turn Agents Training
- MURPHY: Multi-Turn GRPO for Self Correcting Code Generation
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement
- Multi-Turn Code Generation Through Single-Step Rewards
- VERINA: Benchmarking Verifiable Code Generation
- TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code