Rewind to the Fork: Repairing Code at the Token That Decided Its Failure

When a code model's first draft fails its tests, today's repair loops do one of two things. They patch the finished program, editing the lines the error message points at, or they throw the draft away and sample a new one. Neither matches what recent work on reasoning shows about how outputs are actually determined. A handful of "fork" tokens decide most of the outcome. These are the few positions where the model picks a strategy, a data structure, or a loop bound, and everything after them follows almost mechanically. This proposal tests a third option. Locate the earliest fork whose alternative choice turns failure into success, using the interpreter as the judge. Keep the draft up to that token, swap the token for an alternative, and let the model regenerate everything after it. We call this a rewind.

The bet is that rewinding explains, and partly cures, two well-documented failures of iterative coding agents. The first is the plateau, where refinement gains stop after three to five rounds. The second is erosion, where code grows more tangled and verbose with each round. Both would follow if the decisive mistake usually sits upstream of where patches are applied. Each patch would then have to work around a wrong early commitment instead of undoing it. The model would also stay loyal to its own prior output, which is a known tendency.

Core Claim

We claim that, for a large fraction of failing drafts, there is an early fork token at which a small set of the model's own top alternatives flips the program from failing to passing on visible tests. A refinement operator that rewinds to that token should beat both in-place patching and fresh resampling at equal compute. Its advantage should be largest exactly where patching plateaus.

The claim has two parts, a diagnostic one and a method one. The diagnostic part asks where the causal forks are in failing code: how early they occur, and whether they show up as high-uncertainty tokens. The method part asks whether rewinding to them helps once its cost is counted. Either part could fail on its own. A clear negative answer would also be informative, for example finding that fresh resampling matches rewinding at equal compute.

What the literature establishes, and what it leaves open

Three papers motivate the diagnostic. Sparse but Critical studies static math reasoning rollouts offline. It finds that reinforcement learning changes fewer than 4% of token positions, mostly by re-ranking the base model's top-3 candidates. Swapping that small set into a base model moves AIME 2024 accuracy from about 8% to about 25%, while reverting 5% collapses an RL model back to base level. Beyond the 80/20 Rule shows that training only on the top 20% of tokens ranked by entropy (the spread of the model's next-token distribution) improves Qwen3-32B by 11.04 points on AIME '25. ACToR finds that only 5–11% of tokens in repository-level code are critical, mostly identifiers and keywords. It re-decodes them during a single generation pass with retrieved context.

None of these work inside an execution-feedback loop. The first is purely forensic. The second is a training method. The third intervenes during one attempt, without tests. We borrow their "selection, not invention" observation as a design constraint: alternatives at a fork come from the model's own top few candidates.

One paper constrains the design. SIRIUS rolls back and corrects low-probability tokens from a sparsified model and recovers dense GSM8K accuracy by correcting about 11% of tokens. It also reports that error-inducing tokens are often generated confidently. In code this is plausible: a wrong < versus <= can be a near-certain token. Entropy alone therefore cannot be the locator. Execution has to decide which forks matter, and entropy can at most propose candidates. Our method also differs from SIRIUS in what triggers the rollback. SIRIUS rolls back on a dense model's disagreement in fixed chunks. We roll back on a test failure, across refinement rounds.

Four papers establish that the problem is real. InterCode shows large gains from "try again" execution loops, with GPT-4 on SQL going from 9.1% to 73.7%, followed by plateaus and repeated errors. CATArena shows that agents often discard working logic and oscillate across rounds ("negative evolution"). SlopCodeBench shows that structural erosion rises in 77% of agent trajectories and that quality-focused prompts do not stop it. Work on hallucination snowballing shows that models side with their own earlier false claims even against direct evidence, with accuracy drops of at least 31%. Together these suggest a mechanism for patch-based repair: the buggy program stays in context, and the model defends it.

Existing repair methods all act on the finished program as a whole. RePair trains a repair actor with process-based rewards. Self-Edit uses a small fault-aware editor. Stream of Revision revises spans during decoding, supervised on security diffs. ReflectionCoder distills repair trajectories into one-pass generation. None asks at which generation step the draft went wrong. The context-rewriting systems ACR and LLM-as-RNN do remove poisoned history, but they rewrite natural-language state in dialogue and forecasting. Rewinding is the code-generation analogue: it deletes the poisoned suffix while keeping the verified-good prefix.

Research Program

The first phase is a fork census. We take failing first drafts from an open-weights code model, for which we have token probabilities. For each draft $y = (y_1,\dots,y_L)$ we select candidate positions $t$, and for each we substitute an alternative token $a$ from the model's top-3 at that position. We then regenerate the rest of the program greedily and run the visible tests. The fork score is

$$\Delta_t = \max_{a \in \text{top3}(t),\, a \neq y_t} \; \text{pass}\big(y_{<t} \oplus a \oplus \text{continue}\big) - \text{pass}(y),$$

where $\text{pass}(\cdot)$ is the fraction of visible tests passed and $\oplus$ denotes concatenation followed by greedy continuation. The causal fork is the earliest $t$ with $\Delta_t > 0$.

Probing all positions is too expensive. The census therefore probes the union of three candidate sets: the top-entropy positions, positions where the gap between the top-two probabilities is small, and tokens on lines touched by the traceback or failing assertion. On a subset we additionally run exhaustive probing to measure what that union misses. The census reports four things: where causal forks fall in the program; what fraction have low entropy (the "confidently wrong" rate); whether forks fall upstream of the lines a traceback names; and whether drafts whose fork is early are the ones on which patch-based loops plateau. The fourth result is the direct test of our explanation for the plateau.

The second phase builds the rewind operator into a standard refinement loop. On failure, the agent runs the fork search under a fixed probe budget and keeps the best rewound continuation. If no fork flips any test, it falls back to a normal patch turn. Rewinding is applied to the agent's own current program at every round, so the loop can rewind a draft that was itself the result of an earlier rewind.

The third phase removes most of the probing cost. We train a lightweight probe, an MLP on hidden states plus entropy features in the style of ACToR's discriminator, to predict census-labeled causal forks. Probing is then spent only on the probe's top few positions.

One caution applies throughout. Forks are scored only on visible tests, and all results are reported on hidden tests. This keeps the method from simply searching for code that passes the tests it can see.

Baselines

The naive reference is the plain "try again" loop from InterCode. The model sees the full history of attempts and execution feedback and writes a new program, with nothing removed. It is the default practice and the regime where plateaus were documented. The file's reported number for this regime is GPT-4 at 73.7% success on InterCode-SQL. The file gives no reported InterCode-Python number for open models, so we will run this baseline ourselves.

The strongest existing repair systems in the file are RePair and Self-Edit. RePair reports 44.34% pass@1 on CodeNet4Repair with a StarCoder-15.5B actor, against GPT-3.5 at 46.39%. Self-Edit reports raising GPT-3 pass@1 on APPS-dev from 26.6% to 32.4%. Both are the strongest competitors because they are trained specifically to turn execution feedback into a better program. We compare on CodeNet4Repair and APPS-dev against these published figures. We also apply the rewind operator on top of each of them as a backend, since rewinding is meant to complement learned editors rather than replace them. Beating RePair means exceeding 44.34% pass@1 on CodeNet4Repair at comparable actor size and an equal number of test executions.

The comparison most likely to sink the idea is equal-compute resampling. This means best-of-N with the same number of generated tokens and test executions, plus "restart with feedback," where a fresh draft is generated from the problem and a short summary of the error, without the old code. This baseline captures every benefit of rewinding that comes merely from escaping the buggy context, so rewinding must win against it.

Three ablations isolate the key ingredient, which is execution-located forks. Random-position rewind keeps the budget and the top-3 alternatives but picks positions uniformly. Entropy-only rewind picks the highest-entropy positions without test probing, which is in the spirit of SIRIUS and HighEnt. Rollback-to-previous-round reverts to the last passing-more-tests version, in the spirit of the checkpointing in Fang et al..

All methods share the base model, prompts, visible/hidden test split and sampling temperature. They are compared at matched budgets of total generated tokens and test executions, reported as curves over budget rather than at a single point. Each method gets the same hyperparameter-search allowance on a held-out development split.

Why not just…?

Why not just restart from scratch with the error message? This is the real threat. If the benefit of rewinding is only that it escapes contextual inertia, then a fresh sample conditioned on a feedback summary gets the same benefit for free. The diagnostic predicts where the two should differ. Rewinding should win when the causal fork is late, because the correct prefix is expensive to rediscover by sampling. Restarting should do about as well when the fork sits in the first few tokens. We report both methods stratified by the position of the causal fork. If restart matches rewind across all strata at equal budget, we will report that the location of the fork is not useful for repair.

Why not just ask the model to output a minimal diff instead of regenerating? Diff-style patching is what RePair-, Self-Edit- and Stream-of-Revision-style systems already do. Our hypothesis is that patching fails precisely when the wrong commitment sits upstream of the edited lines, because the patch must preserve it. The census test of whether plateaus concentrate on drafts with early forks settles this directly. If patching rescues early-fork drafts as often as late-fork ones, our account of the plateau is wrong.

Why not just use entropy to find the forks and skip test probing? It would be far cheaper. However, SIRIUS reports that error-inducing tokens are often generated confidently, and Sparse but Critical reports that human-plausible substitutions can derail long derivations. The census measures the low-entropy fraction of causal forks in code, and the entropy-only ablation measures what that miss costs in pass rate. If entropy-only matches execution-located rewind, the cheaper method wins and phase three becomes unnecessary.

Why not just do best-of-N or tree search over the whole program? Unguided search spends its budget evenly across positions, most of which are deterministic filler. ACToR reports that targeted re-decoding at critical tokens is about 2x faster than iterative regeneration on repository tasks. Self-Edit reports about 12 seconds versus over 110 for reranking on APPS-dev. Rewinding is a form of tree search; the point is that branching happens at the few positions the census marks as causal. The equal-budget curves show whether that targeting pays off.

Proposed Experiments and Decisive Checks

The census runs on InterCode-Python, which is derived from MBPP. It also runs on the execution-feedback settings of ConvCodeWorld, whose 1,140 BigCodeBench problems come with compilation and execution feedback; the file reports no per-model pass rates for it. We use an open code model in the 7–15B range so that probabilities are available and probing is affordable.

The main refinement comparison adds CodeNet4Repair and APPS-dev for contact with RePair and Self-Edit. The primary metric is hidden-test pass@1 after a fixed number of rounds, plotted against compute. Secondary metrics are the slope of improvement over rounds, computed in the style of CATArena's $S_{evo}$ (the least-squares slope of score against round), and the oscillation rate, meaning the fraction of rounds that undo a previously passing test.

Erosion is tested on SlopCodeBench, where the best agents pass about 14.8% of checkpoints end-to-end. We measure its verbosity and structural-erosion scores per checkpoint, for rewind-based versus patch-based agents. Because rewinding regenerates the suffix rather than layering edits onto it, it should grow more slowly on both measures at equal correctness. Rewinding across checkpoints is restricted to the code written for the current checkpoint, so earlier verified behavior is never discarded.

Three results would be decisive. The first is the census: causal forks should be found in a substantial share of failing drafts, and drafts with early forks should be overrepresented among patching plateaus. If either fails, the explanation for the plateau is wrong. The second is that execution-located rewind beats restart-with-feedback and random-position rewind on the equal-budget curve, with the gap concentrated on late-fork drafts. Without this, fork location is not doing the work. The third is that the learned probe recovers most of the gain of full probing at a fraction of the executions. Without this, the method stays a diagnostic tool rather than a practical repair operator.