When a coding agent submits a patch to a real repository, the standard verdict is binary: the repository's tests pass or they do not. That verdict works well when averaged over hundreds of tasks. On any single patch it is often wrong. UTBoost re-checked SWE-bench leaderboard submissions and found 345 "passed" patches that were incorrect. Separately, Investigating Test Overfitting on SWE-bench showed that agents refining against their own tests drift further toward patches that satisfy the assertion and break something else. This is the code version of a gap that natural-language evaluation has long documented. Metrics that rank systems reliably in aggregate become unreliable on individual outputs. Instance-level scoring is the harder and more useful test (Peyrard et al.), and it is the score that matters when the verdict is used as a reward or a filter (Bhandari).
We propose to attach to every test verdict a measure of how much evidence it carries, computed from the program's dependency structure. The measure is the fraction of the patch's downstream effects that actually reach a test assertion. We pair it with a second check that needs no human reference solution: run the original and the patched repository side by side on inputs that exercise code depending on the changed lines, and flag any place where their behavior differs. The core idea fits in two sentences. A passing test only vouches for the parts of the patch whose effects flow into what the test checks, and we can compute those parts by tracing dependencies. Everything else the patch touches can be checked against the unpatched repository, which serves as an oracle for behavior that was supposed to stay the same.
Core Claim
We claim that false passes and overfitted patches cluster in a measurable region: patches whose downstream effects are mostly unchecked by assertions, or whose behavior on dependent code diverges from the original repository. Two quantities computed from a dependency slice should identify those patches at the instance level, without the gold patch that UTBoost requires.
The first quantity is checked slice coverage. A dependency slice is the set of statements that can influence, or be influenced by, a given statement through data flow (a value computed here is used there) or control flow (this branch decides whether that line runs). Let $S_\Delta$ be the forward slice of the statements a patch modifies. Let $A$ be the set of statements that are both executed by the passing tests and lie on a backward dynamic slice from some failing-then-passing assertion. Checked slice coverage is
$$C(\Delta) = \frac{|S_\Delta \cap A|}{|S_\Delta|},$$
the share of the patch's reachable consequences that some assertion actually observed. This differs from ordinary coverage, which counts lines that merely ran, whether or not their results were checked. A patch can be fully line-covered with $C$ near zero.
The second quantity is a set of divergence witnesses. These are concrete inputs to functions in $S_\Delta$ but outside the issue's focal region on which the original and patched repositories return different values or raise different exceptions. Some divergence is intended, because fixing a shared helper legitimately changes its callers. We therefore do not count witnesses raw. Each witness is shown to a judge model as a triple (input, old behavior, new behavior), together with the issue text. The judge is asked only whether this specific change is explained by the issue.
The novelty is in what plays the oracle role. UTBoost compares candidate patches against the human gold patch. We compare against the pre-patch repository, restricted by the dependency slice to behavior the issue did not ask to change. That makes the check usable where no gold patch exists: in deployment, as a reranker, and as a training reward.
How This Relates to Prior Work
Several papers motivate the gap. Beyond UTBoost and the overfitting study, Re-evaluating Automatic LLM System Ranking shows that automatic judges keep Spearman's $\rho \approx 0.95$ across dissimilar systems. Their accuracy drops by up to 25 points in Controllable Kendall's Tau on close calls, and close calls are exactly the regime of choosing between two test-passing patches.
Other papers use code graphs, but for generation rather than judgment. SynFix builds a repository relation graph of files, classes, functions, imports and calls to coordinate multi-file edits. Removing that graph and its dynamic analysis drops SWE-bench Lite resolution from 52.33% to 31.00%. However, SynFix still validates patches with the repository's existing tests. The Working Set of a Coding Agent formalizes the facts an edit depends on as a coupled-fact graph. It shows that withholding a coupled fact degrades results linearly, but it studies agent behavior on hand-built graphs and does not score candidate patches. We reuse SynFix's style of repository graph to find cross-file dependents. We then turn the coupled-fact intuition into a measurement on real patches.
Three papers are the direct competitors on scoring. CodeBLEU adds data-flow graph matching to BLEU. It needs a reference snippet and works on single functions, whereas we are reference-free and repository-wide. The SWE-Gym verifier ranks trajectories as flat text and is trained on test pass/fail labels, so it inherits the false passes it should catch. Adversarial Review forces critic agents to cite code. It reports that ungrounded critics drift into speculative "scope creep." Our judge sees only concrete divergence witnesses inside a computed slice, which is a direct test of whether grounding the critic in execution removes that failure.
Reps and Yang and Ramalingam and Reps supply the semantics: a slice captures everything that can affect a computation. That is why agreement on the non-focal slice is a meaningful invariant.
Research Program
We build the tool for Python repositories in SWE-bench and SWE-Gym. Static program dependence graphs are fragile in dynamic Python, a limitation the field recognizes, so we avoid relying on them alone. Statement-level dependence is computed dynamically by tracing test execution. The static relation graph serves only to enumerate cross-file callers and importers of modified functions that the tests never reached. The static part says where to look; the dynamic part says what flowed where.
The focal region of the change is defined as the backward slice of the fail-to-pass assertions in the patched program. The non-focal forward slice is $S_\Delta$ minus the focal region.
Inputs for divergence testing come from three sources. We replay arguments recorded from existing pass-to-pass tests that enter non-focal functions. We use type-guided mutation of those arguments. We also generate inputs with a language model prompted with the function and its slice context. Every input is run in both containers, original and patched, and outputs, exceptions and visible side effects are compared.
The final instance score combines four signals: the test verdict, $C(\Delta)$, the count of judge-unexplained witnesses, and optionally the SWE-Gym verifier score. A small logistic model fits the combination on held-out labels. The point is calibration, not a large learned model, because we want the evidence to stay inspectable.
A secondary analysis asks whether $C$ alone explains UTBoost's false passes. If false passes are almost uniformly low-$C$, then the cheapest fix for benchmark integrity is to report $C$ alongside resolution rates, and the divergence machinery becomes optional.
Baselines
The naive reference point is the standard SWE-bench verdict, fail-to-pass plus pass-to-pass tests. It is evaluated on the same leaderboard patches. By construction it labels every false pass as correct, so it fixes the precision floor for false-pass detection.
The strongest reference-free competitor on reranking is the SWE-Gym outcome reward model (SWE-Gym). With a fine-tuned Qwen-2.5-Coder-32B policy it reports Best@16 of 32.0% on SWE-bench Verified, against Pass@1 of 20.6% and an oracle Pass@16 of 42.8%. We will use the released policy samples where available, or regenerate 16 candidates per task with the same policy. We rerank with the SWE-Gym verifier alone and with the verifier plus our signals. Selection compute is matched by counting judge tokens against a verifier given an equal token budget for chain-of-thought scoring. Beating it means a Best@16 above 32.0% at matched selection cost. The 42.8% gap is the room we aim to close.
The strongest method on detecting false passes is UTBoost itself (UTBoost). It found 345 false passes using gold-patch comparison. The file reports no precision or recall for UTBoost as a detector, because its findings are the labels. We therefore treat UTBoost's labels as ground truth. We build a reference-free UTBoost-style baseline: its LLM test generation, with the gold-patch oracle replaced by the same original-versus-patched comparison we use, at an equal number of generated tests and an equal model budget. Beating it means higher area under the ROC curve for flagging the 345 false passes among test-passing patches.
The text-judge baseline is a strong LLM judge given the issue, the diff and the retrieved files. It is scored with the same model we use for witness adjudication, so the only difference is what the judge sees.
The ablations isolate the key ingredient. We replace $C$ with plain changed-line coverage. We replace slice-targeted inputs with inputs drawn uniformly from all functions in modified files. We drop the witness judge and count raw divergences.
Why not just…?
Why not just run more of the existing tests, or the full suite? Pass-to-pass tests already run the rest of the suite, and the 345 false passes survived it. The overfitting study reports that even with golden test access, resolved instances rose only about 3%, so better tests at the focal site do not address collateral damage. Our divergence check targets exactly the code those tests do not assert on.
Why not just use ordinary code coverage of the diff? This is the most serious threat. Investigating Test Overfitting already found that unbiased patches have higher coverage, so coverage carries real signal. Our claim is narrower: lines that ran but whose values never reached an assertion should not count. The changed-line coverage ablation settles it directly. If plain coverage matches $C$ in AUROC, we will report that the dependency tracing is unnecessary for the detection task and keep only the divergence component.
Why not just ask a strong LLM judge to review the patch? Judges collapse on close calls (Re-evaluating Automatic LLM System Ranking). They rubber-stamp or invent concerns (Adversarial Review), and they follow repository documentation even when it contradicts working code (The Working Set of a Coding Agent). We still use a judge, but only on concrete behavioral witnesses. The text-judge baseline, at the same model and token budget, measures whether that restriction matters.
Why not just use UTBoost? It needs the human gold patch, so it cannot score new patches in deployment or serve as a training reward. Our reference-free UTBoost-style baseline tests whether its LLM test generation alone, without slice targeting, is enough.
Proposed Experiments
The first experiment is false-pass detection on SWE-bench Lite and Verified leaderboard patches that pass the official tests, using UTBoost's corrected labels as ground truth. We report AUROC and precision at a 10% flag rate for every baseline and ablation. We stratify by $C$ to test the central prediction: the false-pass rate should fall monotonically as $C$ rises, and most false passes should fall in the lowest third of $C$.
The second experiment is overfitting detection on TDD-bench Verified (449 instances). We use the GPT-4o and Claude-3.7 Sonnet refinement settings from the overfitting study, where reported overfitting rates are 35.9% and 25.5% after refinement. We measure how many overfitted patches our score flags. Then we feed the flags back into the refinement loop to see whether overfitting falls below those reported rates without lowering resolution.
The third experiment is Best@16 reranking on SWE-bench Verified against the SWE-Gym verifier's 32.0%, at matched selection compute.
A fourth check addresses scope creep. Using SWE-PRBench diffs, where Adversarial Review reports an F1 of 0.533, we test whether witness-grounded critiques produce fewer speculative objections than ungrounded ones. A speculative objection is one that cites no executed behavior.
The decisive outcome is the comparison between checked slice coverage and plain coverage in the first experiment, together with any reranking gain over 32.0% in the third. If both favor the slice-based signals, dependency structure earns its place in instance-level code evaluation. If plain coverage ties, the contribution reduces to the reference-free divergence oracle, which we would then report on its own.
References
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- Investigating Test Overfitting on SWE-bench
- Learning to Score System Summaries for Better Content Selection Evaluation
- Re-evaluating Evaluation in Text Summarization
- Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference
- SynFix: Dependency-Aware Program Repair via RelationGraph Analysis
- The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- The Semantics of Program Slicing and Program Integration
- Semantics of program representation graphs