Learning Where Bugs Really Live from Fixes That Got Fixed Again

Models that locate bugs, a task called fault localization, have a known habit: they point at the line where the program crashed instead of the earlier line that actually corrupted the data. This proposal argues that part of that habit comes from the training labels. Those labels are usually taken from the fix commit that closed a bug, and some of those fixes were themselves wrong. They patched the symptom and were later reverted, reopened, or followed by a second fix elsewhere. Software history keeps a correction record, much as the scientific literature has errata and retractions, and nobody uses it to clean fault localization training data. We propose to use it. A first fix that was later superseded becomes a labelled example of a tempting wrong location. The final fix becomes the root-cause label. A small, correction-consistent training set built this way should teach a model to trace backward rather than stop at the crash.

The opportunity matters because localization, not patch writing, is the bottleneck. When models are handed the true faulty file, resolution rates rise sharply: Claude 2 goes from 1.96% to 4.8% on SWE-bench, and GPT-4 Turbo goes from 0.72% to 5.38% on Linux kernel crashes in kGym. Even strong agents mostly localize the crash site. In FaultLoc, GPT-5.3-Codex reaches only 24.90 line-level F1. Its recall of the true code blocks falls from 55.8% on single-site bugs to 24.3% on bugs whose fix spans several places.

Core Claim

Fault localization training labels are noisy in a specific, structured way: a fraction of them are symptom-site patches that a later correction replaced. Two steps should reduce the symptom trap more than generic quality filtering does. The first is to identify these instances from the repository's own correction history. The second is to train on the final, correction-consistent location while treating the superseded first-fix location as an explicit negative. Generic filtering here means keeping only verified patches, or scoring examples by embedding-based data value. The effect should be largest on multi-site bugs and on the "symptom rate," the share of predicted lines that sit at the sanitizer's top crash frame.

The same idea applies to scientific peer review. A reviewer concern that comes back in a later review round marks a revision that addressed the symptom rather than the cause. We test this second setting as a transfer case rather than treating it as the main contribution.

How This Differs from Existing Work

Three bodies of work motivate or supply pieces of this proposal. None of them combines correction history with localization labels.

The first supplies the curation lesson. Co-PatcheR trains a 14B localization-and-patch model on 6,000 strictly correct distilled trajectories and reaches 46% on SWE-bench-Verified, beating systems trained on 11 million uncurated samples. SHED shows that a Shapley-scored subset can beat full-data instruction tuning: a 4,000-sample subset of WizardLM scores 35.92% against 33.16% for the full 70,000 samples. Both define quality as correctness, or as marginal value under a text embedding. Neither asks whether a correct-looking label points at the root cause. Co-PatcheR discards negative trajectories outright. We instead turn a specific class of negatives, superseded first fixes, into localization supervision. This addresses the tension between needing failure examples to diagnose bugs and avoiding them when generating patches. Negatives only shape where the model looks; they are never used as text the model learns to generate.

The second supplies the lens. Scientific integrity research treats the record as cumulative and correctable. Errors persist during a "window of risk" before correction, and downstream work builds on them in the meantime. Assessing the Impact of Retracted Articles on the Reliability of Citing Literature argues that binary retraction flags are too crude. A citation to a retracted paper can be a criticism of it, which makes that citation part of the correction rather than a hazard. We carry this distinction into code. A superseded fix is not simply bad data to delete, because the pair of first fix and final fix records exactly the symptom-versus-cause contrast we want the model to learn.

Retractions relevant to COVID-19 finds that over 30% of retraction notices give no reason. It constrains our design. Revert commits and reopened issues are likewise often unexplained, so correction detection cannot rely on commit messages alone and must use structural evidence. Treat societally impactful scientific insights as open-source software artifacts proposes treating papers as version-controlled software but builds no localization algorithm. We take its analogy in the opposite direction and use the record-integrity view to repair software training data.

The third supplies the target task and the transfer setting. FaultLoc supplies the benchmark and the documented symptom trap. AutoSupervision supplies the transfer setting. It grounds whether a revision resolves a reviewer concern, and it reports that its own data lack negative cases. The recurring-concern signal is a source of such negatives that its corpus already contains.

Research Program

The first component is a correction miner. For C/C++ memory bugs, it uses the reproducible OSS-Fuzz history that FaultLoc itself draws on, restricted to projects disjoint from FaultLoc's 124 evaluation projects. For Python, it uses the 19,000 SWE-bench-train instances.

For each fixed bug, the miner searches forward in history for a correction event of one of four kinds:

  • a revert of the fix;
  • a reopened or re-reported issue, or a regression of the same fuzzer crash signature, after the fix;
  • a later commit that references the same bug identifier;
  • for fuzzing bugs, a new crash reproduced from the same proof-of-concept input on a later version.

When such an event exists, the miner records the first-fix region and the final-fix region. The first-fix region is the code that the first fix touched. The final-fix region is the code touched by the fix that held. Following the retraction-reason lesson, each event is labelled by its structural evidence rather than its message. An event also counts as a symptom correction only if the final region lies strictly upstream of the first region along the data flow. We check this with lightweight static dependence analysis.

The second component is the training objective. Let $y$ be the final-fix line set, $s$ the superseded first-fix lines not in $y$, and $p_\theta(\ell)$ the model's probability that line $\ell$ is faulty. We fine-tune on

$$\mathcal{L} = -\sum_{\ell \in y} \log p_\theta(\ell) \; + \; \lambda \sum_{\ell \in s} \log\big(1 + p_\theta(\ell)\big),$$

where the second term penalizes probability mass on the tempting wrong location, and $\lambda$ sets how strongly. For uncorrected instances, $s$ is empty, and only the instances that pass verification are kept. In practice, localization is emitted as text, so $p_\theta(\ell)$ is read from the model's token probabilities over line identifiers. Uncorrected instances also receive a confidence weight that grows with how long the fix survived without correction. This is the code version of a paper that has stood unchallenged through its window of risk.

The base model is a Qwen-family coder at 14B, matching Co-PatcheR, and the localization pipeline is hierarchical in the Co-PatcheR style: first the file, then the lines.

The third component is the transfer to peer review. In AutoSupervision episodes with more than one review round, a concern that the reviewer raises again after a revision marks the first revision as a symptom-level response. The span the authors changed then becomes $s$, and the span changed in the round that settled the concern becomes $y$. We fine-tune Qwen3.5-9B, the model AutoSupervision already fine-tunes, with the same objective.

Baselines

The naive reference is fine-tuning on a uniformly random subset of the mined training pool, matched in size to the curated set. It carries the pool's natural mix of symptom and root-cause labels. If correction-aware curation cannot beat it, the idea fails.

The strongest existing curation method is correctness filtering in the style of Co-PatcheR. It keeps only instances whose fix passes the bug's tests, or whose fuzzer reproducer no longer crashes, and trains on the fix location as the label. This is the method our idea most directly claims to improve on, because it is what practitioners already do and it produced the strongest small-model result in the file: 46% resolved on SWE-bench-Verified (Co-PatcheR).

The strongest general-purpose valuation method is SHED-style Shapley scoring over clustered instance embeddings. We run it on the same pool with the same subset size. Its best reported gains in the file are 35.92% against 33.16% on WizardLM and 49.21% against 45.99% on ARC-challenge (SHED).

The strongest off-the-shelf localizers on the target benchmark are the frontier agents reported in FaultLoc. GPT-5.3-Codex reaches 24.90 line F1, about 74% file accuracy, and 55.8% and 24.3% single-site and multi-site hunk recall; Claude Sonnet 4.6 reaches 29.57 line precision. A classical spectrum-based localizer serves as a non-learned reference. It scores lines by how much more often they run in failing than in passing executions, as formalized in Distributed Spectrum-Based Fault Localization, and we run it from the available reproducers.

Three ablations isolate the key ingredient. The first drops corrected instances entirely rather than using them as negatives, which separates "remove noise" from "learn the contrast." The second keeps corrected instances, relabels them to the final fix, and sets $\lambda = 0$, which tests whether the negative term matters beyond relabelling. The third removes a difficulty-matched random set of uncorrected instances in place of the corrected ones. This guards against the possibility that corrected bugs are simply harder and that removing them makes training easier.

All fine-tuned variants share the base model, the token budget, the number of epochs, and a hyperparameter sweep of equal size. Training data are disjoint from FaultLoc and SWE-bench-Verified projects.

Beating the baselines means two things. Our model must improve multi-site hunk recall and line F1 over correctness filtering at the same subset size, with confidence intervals from three seeds that do not overlap. It must also reduce the symptom rate. Matching GPT-5.3-Codex's 24.90 line F1 with a 14B model would be a strong secondary result, but it is not required.

Why not just…?

Why not just keep only verified patches, as Co-PatcheR does? This is the real threat. A guardrail placed at the crash site often makes the reproducer stop crashing and the tests pass. FaultLoc's description of agents placing narrow guardrails at the crash site suggests exactly that. If so, verification cannot see the noise we target. The experiment settles it directly. We measure what fraction of verified-but-later-corrected instances the verification filter keeps. If that fraction is near zero, or if correctness filtering matches our method on multi-site recall and symptom rate, the claim is false, and we will report it as such.

Why not just use a generic data-valuation method like SHED? Agentic Instruction Data Selection argues that no single quality metric carries across tasks. There is also no reason a sentence embedding of a bug report and diff would separate a symptom patch from a root-cause patch, because the two read alike. The head-to-head comparison at matched subset size tests this. As a diagnostic, we also check whether SHED scores correlate with our correction labels at all.

Why not just prompt the model to trace data flow backward, or hand it a dynamic slice? Prompting and slicing are cheap and could reduce the symptom rate without any training. We run backward-tracing prompts and slice-restricted context on the same base model as additional comparisons. If they close most of the gap, the contribution shrinks to a data-cleaning result, and the note would say so. We expect them to help single-site bugs more than multi-site ones, because one slice from the crash rarely reaches every disconnected location that needs fixing.

Why not just delete corrected instances and move on? That is the first ablation. If deletion matches the contrastive objective, the integrity-inspired framing reduces to "correction history is a good filter." That is still useful, but it is a weaker claim than "superseded fixes are valuable negatives."

Proposed Experiments and Decisive Checks

The first check is whether the phenomenon exists at useful scale. Before any training, we report the fraction of mined fixes with a structural correction event. We also report the fraction of those events whose final fix lies upstream of the first, and the false-positive rate of the miner on a hand-checked sample. If upstream symptom corrections are too rare to supply a training signal, the program stops here and reports that finding. We expect this check to be informative either way.

The main experiment trains each variant from the Baselines section and evaluates on FaultLoc's 900 vulnerabilities. Metrics are file accuracy, hunk recall split into single-site and multi-site bugs, line precision and F1, and the symptom rate computed against sanitizer top frames. The decisive pattern is that the gain over correctness filtering is concentrated in multi-site recall and the symptom rate. A uniform small gain everywhere would point to generic data improvement rather than our mechanism.

The second experiment checks whether better localization helps downstream repair. We plug the localizer into a Co-PatcheR-style pipeline on SWE-bench-Verified and compare against Co-PatcheR's reported 46% resolved. We also run the oracle-file upper bound to measure how much of the localization gap is recovered.

The third experiment checks generalization across languages and codebase sizes. We evaluate the C/C++-trained model on kGym kernel crashes, using top-3 buggy-file retrieval against the reported BM25 rate of about 10%.

The fourth experiment is the peer-review transfer, on AutoSupervision grounding. The reported references are a grounding score above 0.70 for top models and 0.085 for GPT-4o-mini. AutoSupervision also reports that SFT of Qwen3.5-9B improves grounding, but the file contains no exact number for that model, so no reported result was found on Lacuna for Qwen3.5-9B grounding after SFT. We will rerun that SFT as the baseline. The check is whether recurring-concern negatives improve grounding and verification on concerns whose first revision was insufficient. That subset is where AutoSupervision's lack of negative cases should hurt most. If the gain appears in code but not in peer review, we will report the correction-record idea as specific to software rather than general.