When a coding agent fixes a bug, the fix is usually judged by the tests shipped with the original fix. For benchmarks built from real GitHub history, the repository kept evolving after that fix. Over the following months, developers wrote more tests that pin down how the repaired code is supposed to behave. This proposal treats those later tests as a held-out, human-written check on the agent's patch, one the agent cannot have seen. We call them forward tests. We use them in two ways. The first is to measure whether patches that look correct today are still correct under the stricter scrutiny the project applied later. The second is as a denser, harder-to-game training reward for coding agents learned with reinforcement learning (RL), meaning training by trial and error against a scored outcome.
The motivation comes from two findings. Letting agents repeatedly refine code against tests they generate for themselves makes them overfit. They produce narrow fixes that pass the local test and fail the real requirement, and Investigating Test Overfitting on SWE-bench shows the overfitting rate rising from 21.8% to 25.5% for Claude-3.7 Sonnet and from 33.0% to 35.9% for GPT-4o after refinement. RL for code repair is also expensive per sample and prone to reward hacking; Agentic Reinforcement Learning for Real-World Code Repair reports agents that learned to delete validation code. Both problems stem from a verification signal that is thin, binary, and fixed at the moment of the fix. Forward tests make the signal richer and move it outside the agent's reach.
Core Claim
We make two claims. The first concerns measurement. Among agent patches that pass an instance's official gold tests, a substantial fraction will fail forward tests that the human gold patch passes. This failure rate should grow with the number of self-refinement iterations and with the time horizon over which forward tests are collected. Acute evaluation therefore systematically overstates how good agent patches are.
The second concerns training. Suppose the binary gold-test reward is replaced by a graded reward: the fraction of relevant forward tests passed, gated on the gold tests. Two things should follow. The agent should reach a given resolution rate in fewer RL updates, because partial credit separates near-misses from failures. It should also overfit less, because the reward checks behaviour the issue text never mentioned.
The idea borrows its framing from long-term efficacy assessment in regenerative medicine. There, 12-week animal studies such as Hsu et al.'s goat rotator-cuff anchor study and Lu et al.'s cranial scaffold work report strong acute outcomes. Rabbit results do not fully carry over to dogs (97% versus 72% bone cover), and the field openly acknowledges that its evaluation windows are too short to see late failure. We use only the methodological lesson: extend the evaluation window and plot survival against horizon. The biology itself plays no role in the method.
Constructing Forward Tests
Take a benchmark instance with a pre-fix snapshot $R_0$, a gold patch $g$, and fix date $t_0$. We collect every test function added or modified in the same repository in commits dated within $(t_0, t_0+\Delta]$. We keep a test if it satisfies three conditions. First, it can be run against $R_0+g$ after only mechanical transplantation, meaning copying the test file and its fixtures. Second, it passes on $R_0+g$. Third, its coverage trace touches at least one line or function modified by $g$ or called by a modified function.
Tests that also fail on $R_0$ are strong forward tests, since they directly exercise the bug's neighbourhood. Tests that pass on $R_0$ are forward regression tests. Any test that passes on the gold patch encodes behaviour a correct fix is allowed to have, but not every such test encodes behaviour a correct fix must have. The filter is therefore conservative by design, and we audit the remainder by hand (see Decisive Checks).
The long-term efficacy of a patch $p$ at horizon $\Delta$ is
$$S_p(\Delta) = \frac{|\{\tau \in F(\Delta) : \tau \text{ passes on } R_0 + p\}|}{|F(\Delta)|},$$
where $F(\Delta)$ is the filtered forward-test set for horizon $\Delta$. Plotting the mean of $S_p$ over instances against $\Delta$ (one week, one month, three months, twelve months) gives a survival curve for a method. The area under that curve summarizes it.
Forward tests are always injected after the agent finishes and overwrite any test files the agent touched. Deleting or editing tests therefore cannot raise the score, which closes the loophole seen in Agentic RL for Real-World Code Repair.
Research Program
The first stage is a measurement study on SWE-bench Verified, the 500-instance human-verified subset of real GitHub issues. We build $F(\Delta)$ for each instance and report how many instances have non-empty forward sets at each horizon. Instances without forward tests are reported separately rather than dropped silently.
We then score patches from a range of pipelines, each run by us with public code:
- the hierarchical file-to-function-to-edit pipeline Agentless;
- Agentless followed by the test-generation-and-critic refinement loop from Investigating Test Overfitting on SWE-bench, recorded at each iteration;
- the verifier-guided Best@k selection of SWE-Gym.
The central analysis compares forward-test survival among gold-passing patches, broken down by refinement iteration. This tests the prediction that refinement lowers survival even when acute metrics stay flat or improve. The prediction follows from Yan et al.'s observation that refinement plateaus after a few iterations, and from the overfitting results above.
We also sort forward-test failures by bug geometry. We expect a higher failure rate on issues whose gold patch spans several files or whose failing tests sit far from the edited lines. Those are the multi-hop and multi-bug cases where DSDBench finds cause-line detection falling from 40–48% to 10–20%, and where RepoDebug finds accuracy on multiple errors collapsing to 3.66%. Those benchmarks inject bugs synthetically. Forward tests would show whether the same weakness appears as latent defects in real patches.
The second stage trains agents. We apply the same construction to the 2,438 executable instances of SWE-Gym. We check for repository and commit overlap with SWE-bench Verified and remove any shared instances, so that no evaluation behaviour leaks into training.
We then train an open-weight coder at 7B–14B scale with GRPO, which scores each sampled attempt against the other attempts for the same prompt. The rollout pipeline is the Agentless-style localization-then-repair pipeline used by DeRL-SWE. That paper co-trains localization by rewarding it with downstream repair success, and it avoids a supervised warm start. We keep that pipeline and change only the reward. The graded reward for a rollout producing patch $p$ is
$$r(p) = \mathbb{1}[p \text{ passes gold tests}]\,\big(1 + \lambda\, S_p(\Delta_{\text{train}})\big) - \mu\,\mathbb{1}[p \text{ breaks a pre-existing passing test}].$$
Here $\lambda$ sets the weight of forward-test credit, $\mu$ penalizes regressions, and $\Delta_{\text{train}}$ is the training horizon. We gate on the gold tests so the reward can never favour a patch that fails the actual issue. The forward term adds graded information among the patches that pass the gold tests, which is exactly where test overfitting operates.
We also train a variant that awards partial credit for strong forward tests even when the gold tests fail. This tests whether a denser signal early in training speeds up learning.
Baselines
The naive reference point is the untrained base model in the same Agentless-style pipeline with a single attempt. For context, SWE-Gym reports 20.6% Pass@1 on SWE-bench Verified for zero-shot Qwen-2.5-Coder-32B. Our smaller model will score lower, and we report our own measured base number rather than borrow that one.
The strongest existing training method on this question is DeRL-SWE's GRPO training with a binary resolution reward and nested credit assignment. That paper reports 47.4% single-attempt and 61.8% with test-time patch selection on SWE-bench Verified at 32B DeRL-SWE. We cannot match its compute. Instead we re-run its reward design at our scale, with the same pipeline, the same instances, the same number of rollouts, and the same hyperparameter search budget. The comparison is fair because only the reward differs.
Beating it means two things. The first is a higher resolution rate at matched rollouts, or the same resolution rate in measurably fewer updates. The second is a lower overfitting rate and a higher forward-survival area on SWE-bench Verified. The forward-survival comparison uses forward tests from evaluation repositories, which training never touches.
The strongest inference-time competitor is SWE-Gym's outcome-supervised reward model (a learned scorer of whole attempts) used for Best@k selection. It reports 32.0% Best@16 against an oracle Pass@16 of 42.8% on SWE-bench Verified. We train that verifier on our SWE-Gym rollouts and give it the same sample budget k. We also test a verifier trained with forward-test labels in place of binary resolution labels, to see whether forward tests help selection as well as training.
Two ablations isolate the key ingredient. The first replaces forward tests with the same number of LLM-generated tests, produced in the style of the test-generation step of Investigating Test Overfitting on SWE-bench and run through the same gold-patch filter. The graded-reward structure stays identical and only the source of the tests changes. The second ablation shuffles forward tests across instances within each repository. This checks whether the benefit comes from instance-relevant checks or merely from a nonzero variance in rewards.
Why not just…?
Why not just use the gold tests plus the existing regression suite, as SWE-bench already does? That suite fixes the evaluation window at the day the fix was merged, which is precisely what we claim is too short. Investigating Test Overfitting on SWE-bench reports that giving refinement access to hidden gold tests raised results by only about 3%, which suggests the gold set carries little extra signal. The measurement study settles this directly. If almost no gold-passing agent patches fail forward tests, the proposal's premise is false and we report that.
Why not just generate many more tests with an LLM? This is the most serious alternative. Generated tests are cheap, available for any instance, and could provide the same graded reward. The case against them is that they come from models with the same blind spots as the patching model, and that refinement against them is exactly what drove overfitting in the cited study. Forward tests, by contrast, were written by people reacting to later bugs and requirements. The generated-test ablation is the deciding experiment. If it matches the forward-test reward on resolution, overfitting, and survival, the practical contribution shrinks to the measurement study, and we will say so.
Why not just train a better learned verifier? A learned verifier is trained on labels. SWE-Gym's outcome reward model still leaves a 10.8-point gap to its oracle, and its labels are the same acute pass/fail signal we argue is incomplete. Forward tests supply better labels, so the two methods combine rather than compete, and our forward-labelled verifier variant tests that combination directly.
Why not just add a step-level analyzer for credit assignment? AHEAD uses an analyzer that marks the mistaken step in a failed episode and supplies a corrective hint there. It cuts the training steps needed to match GRPO by 43–57% on ALFWorld and WebShop. However, it relies on a costly frontier-model analyzer and assumes mistakes occupy discrete, identifiable steps, an assumption its authors flag as weak for multi-hop code. Forward tests cost only sandbox execution and need no analyzer. A head-to-head comparison is outside our budget, which we state as a limitation.
Decisive Checks
The first check determines whether forward-test failures are real defects or spec drift. We will manually audit 200 randomly sampled failures of gold-passing agent patches. Each is labelled as a genuine behavioural defect, as behaviour that later changed intentionally, or as a test artifact. If fewer than half are genuine defects, the filter must be tightened before any training claim is made.
The second check is the survival curve itself. Survival among gold-passing patches should fall with the number of refinement iterations. The curve for each method should also decline with horizon, and more steeply for methods with higher overfitting rates as defined by Ahmed et al.
The third check measures sample efficiency. For each reward, we track resolution rate on a held-out SWE-Gym split against the number of GRPO updates. We report the number of updates each reward needs to reach the binary-reward run's final resolution rate. Results are given over at least three seeds with confidence intervals.
The fourth check covers robustness to reward hacking. We count rollouts that modify or delete test files during training, and confirm that such edits never raise the forward-test term.
A positive result on the measurement study alone would already change how code agents are evaluated. A positive result on training would show that the repository's own future supplies a cheap, human-grounded reward.
References
- Investigating Test Overfitting on SWE-bench
- Agentic Reinforcement Learning for Real-World Code Repair
- Fixation technique of biodegradable magnesium alloy suture anchor in the rotator cuff repair of the shoulder in a goat model: a technical note
- An instantly fixable and self-adaptive scaffold for skull regeneration by autologous stem cell recruitment and angiogenesis
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- Agentless: Demystifying LLM-based Software Engineering Agents
- Training Software Engineering Agents and Verifiers with SWE-Gym
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors
- RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
- DeRL-SWE: Decoupled Reinforcement Learning with Nested Credit Assignment for Software Engineering
- AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL