Reinforcement learning for code models works by sampling several programs for the same problem, running them against tests, and pushing the model toward the programs that did better than their siblings. When every sibling fails, the group carries no learning signal. Current methods treat this as waste and simply throw the problem away. Those all-fail problems are exactly the ones the model cannot yet solve, and for code they come with something math problems lack: a traceback or failing assertion explaining what went wrong. We propose to spend extra compute on these problems instead of discarding them. When a group's rewards collapse to all-fail, the trainer launches a short burst of execution-guided repair: the model gets its own failed program plus the error output and tries again, several times in parallel. The repair attempts usually succeed and fail in mixed proportions, so the group regains a learning signal, and the model is trained specifically on fixing its own broken code.
The same statistic, how spread out the test outcomes of a group of samples are, then decides compute at deployment. If all candidates pass the public tests, the system stops early. If they disagree, it selects among them. If all fail, it repairs. The claim is that one signal can govern where compute goes in both training and inference. The training half is new. The inference half is a cheaper, learned version of what hand-built test-time pipelines already do.
Core Claim
In critic-free group-relative training such as GRPO, a sample's advantage is its reward minus the group mean, divided by the group standard deviation. Groups with zero reward variance produce zero gradient. Learning More with Less shows that higher advantage variance in a batch corresponds to larger gradient norms. Reward-Guided Prompt Evolving argues that variance is a direct proxy for how useful a prompt is. Both papers motivate our approach, and both respond by selecting or discarding prompts, as do the batch filters in Buffer Matters and the FHIR tool-calling work Reinforcement Learning for Tool-Calling Agents in FHIR.
Our claim is that for code, an all-fail group is better treated as a trigger for conditional repair than as a filter. Let $x$ be a problem and $y_1,\dots,y_G$ the first-turn programs, with hidden-test rewards $r_1=\dots=r_G=0$. We pick a failed program $y_j$ with its execution feedback $f_j$ and sample $M$ repairs $z_{j,1},\dots,z_{j,M}$ from the policy conditioned on $(x, y_j, f_j)$. Each repair $z_{j,m}$ gets a hidden-test reward $s_{j,m}$. Its advantage is normalized only against its siblings that share the same parent:
$$A_{j,m} = \frac{s_{j,m} - \operatorname{mean}_m(s_{j,\cdot})}{\operatorname{std}_m(s_{j,\cdot})}.$$
This measures how good a repair is relative to other repairs of the same broken program, not relative to unconditioned first attempts. That conditioning avoids the early-attempt bias that Learning to Correct (CAL-GRPO) identifies when attempts at different depths are pooled. The prediction we test is specific: at equal sandbox and token budgets, triggered repair recovers informative groups from problems where plain resampling almost never produces a single success. The resulting model also repairs better at test time, so the extra training compute pays for itself in deployment.
Relation to Prior Work
Three papers bear most directly on this proposal, each in a different role.
From Search to Policy (TGPR) is the closest prior work. It uses Thompson-sampling trees during GRPO training to explore repair paths, with a reward that mixes CodeBLEU and unit-test pass rate. At test time it decodes greedily with no search. We differ on three points. Our repair compute is triggered by variance collapse rather than applied to all prompts. We need no reference solution, because the reward comes from tests only. And we keep and exploit test-time repair rather than amortizing it away.
CAL-GRPO supplies our method for normalizing advantages separately at each attempt depth. It uses binary feedback only, a fixed number of attempts, and math and maze tasks. We add traceback feedback, invoke repair only when the variance trigger fires, and work in code.
Re² trains models to abandon and restart failing reasoning. It shows that salvaging flawed premises is often hopeless in math. This constrains our design: repair conditioned on a broken parent may sometimes be worse than restarting. We therefore include fresh restarts as a competing branch inside the repair burst and measure which one wins.
The motivating evidence on the test-time side comes from execution-grounded scaling. S* combines parallel sampling, up to two rounds of traceback debugging, and test-input synthesis to choose between candidates, with frozen weights. Environment Augmented Generation argues that sandbox feedback prunes bad branches before errors compound. Two results warn that repair saturates. The LiveCodeBench self-repair track shows large one-round gains (GPT-4-Turbo from 24.5% to 36.9%). CodeTransOcean reports gains flattening after two to three rounds (48.57% to 52.57% after three rounds). We therefore cap repair depth at two rounds.
Spontaneous Reward Hacking in Iterative Self-Refinement shows that refinement loops game whatever judge they can see. In code, the analogous risk is a model that hard-codes the visible failing test. To guard against this, we show feedback from public tests but compute reward on held-out tests.
MicroCoder-GRPO supplies the single-turn training recipe we build on (truncation masking, diversity-aware temperature, no KL penalty). It has no multi-turn repair.
Research Program
The system has three parts.
The trigger. During each GRPO step, groups with mixed rewards are trained normally. All-pass groups are dropped, as in existing filters. All-fail groups enter a repair burst. We select up to $k$ distinct failed parents, choosing them by diversity of error type (syntax error, runtime exception, wrong answer, timeout) rather than at random, and sample $M$ children per parent. The children include $M_r$ fresh restarts that do not see the parent, so restarts and repairs are compared on equal footing. If a burst is still all-fail after two rounds, the problem goes to a deferred buffer and is retried a few hundred steps later, in the spirit of Buffer Matters.
Credit assignment. First-turn programs in a triggered group receive no gradient, since their advantage is truly zero. Repairs receive the parent-conditioned advantage above. We will also test a variant that gives the first turn a small CAL-style "future value" credit, the empirical repair success of its children. This tests whether learning to produce repairable first drafts helps.
Variance-gated inference. At test time we sample $N$ programs and run the public tests. If all pass and behave identically, we stop. If they disagree, we select using public-test pass counts. If all fail, we run the trained repair policy for at most two rounds. This allocates compute adaptively, unlike S's uniform debugging of every sample. The comparison of interest is whether a trained repairer under gated allocation matches S at a fraction of the sandbox calls.
The base model is Qwen2.5-Coder-7B. S* reports results for this model, and its size matches TGPR's Qwen-7B. Training problems are competitive-programming tasks with test suites, each split into visible and hidden tests. Problems overlapping the evaluation windows of LiveCodeBench are removed by release date.
Baselines
The naive reference is single-turn GRPO that simply discards zero-variance groups, run with the MicroCoder-GRPO recipe and the same problem set. This is what the field does today. Our method must beat it on hidden-test pass@1 at matched training compute. Training compute is counted as total generated tokens plus sandbox executions, not as optimizer steps. Otherwise the repair bursts would get free extra samples.
The strongest training-side competitor is TGPR. It is the only method in the file that trains a code-refinement policy with group-relative RL and search. Its reported result is 41.2 on LiveCodeBench and 77.6 on MBPP for Qwen-7B under greedy decoding, with pass@1 up to +4.4 points over standard GRPO. We will re-run TGPR on our data with its released configuration where possible, give it the same token and sandbox budget, and give it the same tuning budget (an equal number of hyperparameter trials). Beating it means higher pass@1 under greedy decoding, so that test-time compute is not part of the comparison. It also means a larger margin once both models get the same gated test-time budget, which TGPR's design does not exploit.
The strongest test-time competitor is S applied to the untrained base model. On LiveCodeBench, S reports 69.1% to 85.7% for DeepSeek-R1-Distill-Qwen-32B and 61.3% Pass@1 for GPT-4o-mini. For Qwen2.5-Coder-7B the file reports only that S exceeds the 32B baseline by over 10%; no absolute LiveCodeBench number for the 7B model was found on Lacuna, so we will reproduce S ourselves. The comparison is accuracy as a function of sandbox calls and generated tokens. Our method wins if its curve lies above S's at every budget, or matches S's top accuracy with substantially fewer calls.
Three ablations isolate the key ingredient. Repair on all prompts, not only all-fail ones, tests whether the trigger matters. Parent-conditioned normalization versus pooled normalization tests the credit assignment. A repair burst with tracebacks replaced by a bare "failed" message tests whether rich feedback matters or whether this reduces to CAL-GRPO on code.
Why not just…?
Why not just sample more first attempts for all-fail problems, as dynamic-sampling filters do? This is the real threat. If a problem's true pass rate is 3%, drawing 32 extra samples will often produce a success and restore variance without any repair machinery. Our bet is that a substantial fraction of all-fail problems have essentially zero first-attempt success but nonzero repair success. Tracebacks reveal off-by-one errors, wrong input parsing, and timeouts that resampling keeps reproducing. We settle this directly. For each all-fail problem, we measure the probability that a burst becomes mixed under equal-cost resampling versus repair, binned by estimated first-attempt pass rate. If resampling matches repair across bins, the training contribution collapses. In that case we would recast the work as a study of test-time gating alone.
Why not just apply CAL-GRPO with a fixed two attempts on every problem? Uniform multi-attempt training spends repair compute on problems that are already mixed or solved, where one attempt carries enough signal. CAL-GRPO's own analysis shows that later attempts are sampled only on surviving failures, which biases training toward early attempts. The repair-everywhere ablation measures exactly this. If uniform repair matches triggered repair at equal compute, the trigger is not doing work.
Why not just train single-turn and bolt S* on at test time? It is plausible that a stronger first-attempt model plus S beats a trained repairer, since S gives frozen models large gains. The experiment that settles this crosses the two training methods (single-turn versus triggered repair) with the two inference methods (S versus gated repair) at matched budgets. If S erases the training difference, the value of our method lies only in cheaper inference. We would report that honestly.
Why not just restart instead of repair, following Re²? Restarts are included inside every burst, so the policy and the logged statistics reveal which one wins on which error types. We expect repair to win on runtime and parsing errors and restarts to win on wrong-algorithm failures. If restarts dominate everywhere, the traceback component is unnecessary.
Proposed Experiments
Evaluation uses LiveCodeBench code generation, scored on hidden tests over a post-training release window, and its self-repair track. MBPP serves as a check against TGPR's reported numbers, and CodeContests as a harder held-out set. We report pass@1 under greedy decoding, pass@10, and accuracy-versus-budget curves, with the x-axis measured both in sandbox executions and in generated tokens. We also report the training-side statistic that is the heart of the claim: the fraction of zero-variance groups converted into informative groups per unit compute, under repair versus resampling.
Four checks would make or break the proposal. First, the resampling-versus-repair conversion curve must show repair dominating on low-pass-rate problems. Second, triggered-repair training must beat MicroCoder-style filtering and TGPR on greedy pass@1 at matched training compute. Third, gated inference with the trained model must match or exceed S*'s accuracy with fewer sandbox calls. Fourth, the gap between visible-test and hidden-test pass rates must not widen during training. A widening gap would be the code analogue of in-context reward hacking, meaning the model learned to satisfy the feedback rather than the problem. A repository-level follow-up on SWE-bench Lite is deferred, since every method compared here is limited to single-file tasks.
References
- Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
- Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs
- Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
- Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR)
- Learning to Correct: Reinforcement Learning for Multi-Attempt Chain-of-Thought
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
- S*: Test Time Scaling for Code Generation
- a1: Steep Test-time Scaling Law via Environment Augmented Generation
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Spontaneous Reward Hacking in Iterative Self-Refinement
- Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models