Language models can fix their own code in a loop: write a program, run it against tests, read the failures, and edit. Training this loop with reinforcement learning (RL), meaning rewarding the model when its final program passes the tests, is attractive but fragile. Most of the loop is wasted effort. In a study of frontier models, 57% of successful repairs happened on the first edit, and about 45% of failing repair attempts entered "stagnation", where the model kept changing code without passing a single additional test (Rapid Fixes, Gradual Failures). During RL these stalled turns cost sandbox time and tokens. They also corrupt learning. When a trajectory ends in failure, every turn in it, including a good first edit, gets pushed down by the failed outcome.
We propose to detect stagnation directly from execution feedback and cut a training rollout once the model stalls. The cut trajectory is scored by the best program it reached, and the stalled tail gets no gradient. We pair this with a small supervised "warm start", a brief phase of ordinary fine-tuning on example repair trajectories before RL begins. The claim is that these two simple ingredients, together with a plain outcome reward, can replace the hand-tuned reward shaping that current RL-from-scratch code repair systems need to avoid collapse. The trained model should also take fewer and shorter repair turns at inference time.
Core Claim
Our hypothesis is that much of the instability in multi-turn code RL comes from advantage inversion on stagnant tails. The advantage is the per-token signal that says whether an action was better or worse than average. Reducing Belief Deviation in Reinforcement Learning for Active Reasoning of LLM Agents identified this effect in deduction puzzles. When an agent gets stuck in circular behavior, the uninformative end of the trajectory produces negative signals that penalize the useful early actions. Truncating stalled trajectories, a method the authors call $T^3$, restored stable training and raised GRPO by 30.1 points on GuessNumbers. That paper had to infer progress from indirect cues such as repeated questions. In code, progress is directly observable: the set of passing tests and the error the program raises either change or they do not. We therefore expect truncation to transfer to code more cleanly than it did to puzzles.
The same paper adds a constraint. Small 3B models did not benefit from $T^3$, apparently because they lacked a competent prefix worth protecting. This is where the warm start enters. Our second claim is that the benefit of truncation depends on the warm start. Without a minimal supervised anchor, first edits are rarely good, so there is little credit to preserve and truncation should help little. With an anchor of 10 to 25% of the training budget, the size Balancing the Budget found sufficient to fix formatting before preference tuning, truncation should help substantially.
This claim stands in direct tension with ReflexiCoder. ReflexiCoder trains a Qwen3-8B base model with no supervised stage, an approach it calls "RL-zero". It reaches 94.51% single-attempt on HumanEval and 52.21% on LiveCodeBench. To get there it relies on a format-compliance gate, a cycle-count penalty with a sinusoidal term, improvement rewards weighted exponentially over time, and an efficiency bonus. We read the cycle and improvement terms as indirect fixes for the stagnation problem that truncation addresses directly. If the shaping turns out to be doing something truncation cannot, our experiments will show it.
Research Program
The setting is multi-turn repair with execution feedback. At each turn the model sees the problem, its latest program, and the sandbox output, which includes the pass/fail result for each test and a normalized error message. It then emits a revised program or a stop token. Following the finding that full histories can hurt (Rapid Fixes, Gradual Failures), the context holds only the latest attempt, so prompts do not grow with each turn.
A turn is stagnant if three things hold. The set of passing tests is unchanged or a subset of an earlier turn's set. The error signature (exception type plus normalized failing location) matches one already seen. The program is not a strictly better test-passer than any previous program. A rollout is cut after $k$ consecutive stagnant turns, with $k$ in $\{1,2,3\}$ tuned on a held-out split.
Let $p_t$ be the fraction of tests passed at turn $t$ and $\tau$ the truncation turn. The trajectory return is
$$R = \mathbb{1}[\text{all tests pass at some } t \le \tau] + \lambda \max_{t \le \tau} p_t,$$
where $\lambda$ is a small partial-credit weight that we also set to zero in an ablation. Gradient flows only through tokens in turns $1..\tau$, and advantages are normalized within GRPO groups. GRPO is the group-relative policy optimization variant of RL, which compares several rollouts of the same prompt to estimate advantages. Because $\lambda$ creates a dense reward on its own, we separate its effect from the effect of truncation in ablations.
The warm start uses no external teacher, which addresses a known cost problem for warm starts. We prompt the base model with a few examples and run it in the sandbox. We keep only trajectories that end in a passing program and contain no stagnant turns, and we use these for supervised fine-tuning at 0%, 10%, and 25% of the total example budget. The rest of the budget goes to RL.
At inference time, the same stagnation detector serves as a stopping rule. Because the sandbox signal is already computed, the rule adds no model calls.
Two diagnostics run throughout training. The first measures advantage inversion directly. Among turns that strictly increased $p_t$, we track the fraction that received negative advantage. This is the quantity the hypothesis predicts truncation will shrink. The second detects collapse, defined as a drop of more than half in training reward or a sharp spike or drop in policy entropy, the warning signs reported by M-GRPO and ExGRPO. We record collapse across seeds and at two learning rates, one standard and one aggressive at 5× the standard rate, mirroring the stress regime of CAPO.
Baselines
The naive reference point is the base model, Qwen3-8B base to match ReflexiCoder, run with the same few-shot repair prompt and a fixed budget of up to 10 turns with no training. A second naive point is the warm-started model with no RL. Together these show what the supervised anchor alone buys.
The strongest existing method on this question is ReflexiCoder's RL-zero recipe. It is the only cited system that trains a base model for self-correction on code with RL and no supervised stage, and its reported numbers are the bar: 94.51% single-attempt on HumanEval, 52.21% on LiveCodeBench, and 37.34% on Codeforces. Those numbers come from its own pipeline. We will therefore re-implement its reward shaping inside our multi-turn execution-feedback loop, using the same base model, training problems from APPS, number of rollouts, GPU-hours, and an equal hyperparameter search budget for both arms. We will report its published numbers alongside our reproduction. Beating it means matching or exceeding its pass@1 on LiveCodeBench and Codeforces within seed noise, with fewer collapsed runs across seeds and lower mean turns and tokens per solved problem.
A second strong competitor is TGPR (From Search to Policy). TGPR uses Thompson-sampling search trees during training to diversify repair rollouts, and it reports 77.6 on MBPP and 41.2 on LiveCodeBench with Qwen-7B. It requires reference solutions for its CodeBLEU reward and adds 37.5% wall-clock overhead during training. We will compare against its reported numbers and, where budget permits, run a reduced version on the same training data.
Three ablations isolate the key ingredient. The first is truncation with $\lambda=0$, which removes the dense partial-credit reward. The second keeps the dense max-pass-fraction reward but drops truncation, which isolates the effect of removing stalled tails. The third replaces adaptive truncation with a fixed turn cap matched to the mean truncated length, which isolates adaptivity. Each is crossed with the 0%, 10%, and 25% warm-start settings.
Why not just…?
Why not just cap refinement at one or two turns, since most fixes happen on the first edit? This is the most serious threat, and we take it seriously. A fixed cap also removes stalled tails, and on easy benchmarks such as HumanEval, where scores are near ceiling, it may match truncation. It differs from truncation by also cutting productive late turns. The same study that found 57% of fixes on the first edit implies that the other 43% come later, which suggests late repairs matter on harder problems. The matched-length fixed-cap ablation settles this question. If the cap equals adaptive truncation on LiveCodeBench and Codeforces, the mechanism reduces to "train short", and we will report that conclusion rather than the stronger claim.
Why not just use best-of-N parallel sampling, which has no sequential latency at all? Parallel samples avoid waiting on turn after turn. The weakness is coverage: when every first attempt fails, recovery is poor. Rapid Fixes, Gradual Failures reports 22.9% recovery when all seeds fail versus 76.6% when one succeeds. Critique-RL reports up to 5× better compute efficiency for refinement over majority voting, although that result is from math, not code. We will compare best-of-N from the warm-started RL policy against our multi-turn policy at matched sandbox calls and matched wall-clock time. If best-of-N wins at matched wall-clock, the latency argument for multi-turn repair fails for this setting.
Why not just keep ReflexiCoder's shaped reward, or add a curvature-based stabilizer like CAPO? Shaped rewards evidently work, but they carry hand-tuned terms that must be re-tuned for each domain. CAPO masks unstable tokens and has only been shown on math, and its authors expect its thresholds to need re-tuning for code. Our method has one interpretable knob, $k$, grounded in execution. The head-to-head with ReflexiCoder under equal tuning budgets, together with the advantage-inversion diagnostic, will show whether truncation addresses the same failure the shaping addresses. If shaping still adds accuracy on top of truncation, we will report the combination.
Why not just do more supervised fine-tuning on teacher repair traces? Pure supervised fine-tuning plateaus relative to preference or RL tuning in larger data regimes (Balancing the Budget), and teacher traces are costly. The 25% warm-start arm with no RL directly measures how far supervision alone gets.
Proposed Experiments
Training uses APPS training problems. Evaluation covers HumanEval+, MBPP+, LiveCodeBench, and Codeforces. For a compiled-language transfer check, we evaluate on HumanEval-Java and other MBXP languages through amazon-science/mxeval, which addresses ReflexiCoder's Python-only limitation. We run Qwen3-8B base and a 3B model, the latter to test whether the 3B failure seen with $T^3$ recurs and whether a warm start fixes it. Metrics are pass@1 and pass@10, the fraction of runs that collapse (out of five seeds at each learning rate), mean turns and generated tokens per solved problem, wall-clock time per problem including sandbox round-trips, and training tokens consumed.
The first decisive check is the interaction between warm start and truncation. Truncation's pass@1 gain on LiveCodeBench should be near zero at 0% warm start and clearly positive at 10% to 25%, and the effect should be largest for the 3B model. The second is the mechanism check. The fraction of improving turns that receive negative advantage should fall under truncation and track the reduction in collapsed runs. If collapse falls while this fraction does not, the advantage-inversion explanation is wrong. The third is the fixed-cap comparison on hard benchmarks described above. The fourth is the latency check. At matched pass@1, the truncation-trained policy using the stagnation detector as its stopping rule should need fewer turns and less wall-clock time than both the ReflexiCoder-style reward and an untrained 10-turn loop.
As a stress test, we will apply the inference-time stopping rule to Lean 4 tasks from VERINA, without RL training there. VERINA reports that 64 refinement rounds lift proof success only to about 20% while frequently thrashing. We will measure how much of that cost a stagnation-based stop removes and how little success it gives up.
References
- Rapid Fixes, Gradual Failures: Exploring Iterative Self-Correction Dynamics in Large Language Models for Program Synthesis
- Reducing Belief Deviation in Reinforcement Learning for Active Reasoning of LLM Agents
- Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning
- ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
- M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization
- ExGRPO: Learning to Reason from Experience
- Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement
- Critique-RL: Training Language Models For Critiquing Through Two-Stage Reinforcement Learning
- VERINA: Benchmarking Verifiable Code Generation