A learned bug detector, meaning a model that reads a function and predicts whether it contains a defect, works well on the projects it was trained on and poorly on a new one. New repositories have their own naming habits, libraries and idioms, and usually no labeled bug history at all. The standard response is to train on other projects' bugs, or on large synthetic corpora of artificially broken code. Both carry the wrong distribution. Training on the full synthetic corpus can even do worse than skipping it (Learning Defect Prediction from Unrealistic Data). This note proposes a different source of training data. On the target project's own code, we generate candidate bugs and keep only the ones that pass a physical-style admissibility check. The check runs the code: a kept bug must really change program behavior on generated inputs, and it must also pass the project's own developer test suite. In other words, we simulate the specific kind of defect a real detector will face in deployment, namely bugs that are real and that the project's existing tests let through.
The idea is borrowed from physics-informed synthetic data. That field turns cheap random samples into useful training data by passing them through a simulator of the governing mechanics and of the measurement process. A cold-start solar forecaster, for example, can be conditioned on a site's simulated production history before any real data exists (Time series Foundation Models based on Physics-Informed Synthetic Histories for Cold-Start Photovoltaic Forecasting). In our setting, the program interpreter plays the role of the physics. The developer test suite plays the role of the sensor through which real defects are observed, or in this case not observed.
Core Claim
For bug detection on a project with no labeled bugs, synthetic bugs that are behavior-changing, test-escaping and drawn from the target project's own code should close more of the cross-project gap than three alternatives: real bugs from other projects, unconstrained mutants, and embedding-filtered synthetic corpora. Two things matter jointly. Locality to the target removes the style shift. The execution constraint removes two kinds of noise. The first is "bugs" that change nothing, which are equivalent mutants and therefore label noise. The second is bugs that any test would catch, which are unlike the defects that survive in shipped code.
Two pieces of prior work motivate this design. CodeS benchmarks code models under programmer, time and task shifts and finds large degradations, including total failure under task shift. Villmow et al. show that generative code models depend on local project conventions. Two other pieces constrain the design. Hu et al. report that mutation-based augmentation overfits to the source project's style. Alrasheed et al. show that unrealistic synthetic bugs hurt. Together these rule out "more mutants from more projects" as a fix.
The method itself comes from two physics papers. Beyond respiratory models passes random deformation fields through a hyperelastic finite-element solver so that only physically admissible samples remain. That constraint improved real registration error from 3.8 mm to 2.8 mm. Ho et al.'s nuclei segmentation models the instrument's blur and noise, and with that alone it trains a segmenter with no real labels. Our two filters mirror these two ingredients one-for-one. The behavior-change test is the admissibility constraint, and the test-escape requirement is the measurement model.
Research Program
The generator is held fixed so that the effect of the constraint can be isolated. We use the same families of rule-based bug operators that underlie the ETH Py150 synthetic corpus studied by Alrasheed et al.: variable misuse, swapped arguments, wrong binary or comparison operators, and off-by-one boundaries. As a second, richer generator, we add bug rewrites proposed by a code language model. Generation runs on every function of the held-out target project.
Each candidate then goes through two executions. The first is a differential check. We run the original function and the mutant on inputs produced by property-based fuzzing at the function boundary, and on the project's tests, and record any difference in return values, raised exceptions or observable side effects. Candidates with no difference are discarded as non-bugs. The second check runs the developer test suite. Candidates that the tests catch are labeled "shallow" rather than discarded, so that their value can be measured separately. The remainder form the escaped-bug set, and each comes paired with its original function as a clean negative.
The detector is CodeBERT, chosen to match Alrasheed et al. It is fine-tuned on real bugs from other projects and on the target's escaped-bug set, and then evaluated on the target's real bugs. We will also analyze which operator families survive the filter at what rates. We will measure how this survival profile differs between projects. Finally, we will test whether the per-project fraction of escaped bugs predicts how much each project gains, which would give a cheap diagnostic of when the method is worth its execution cost.
Baselines
The naive reference point is CodeBERT fine-tuned only on real bugs from the other projects and applied to the held-out project. This is the ordinary cross-project setting, and every other condition must beat it to justify any synthetic data.
The strongest existing method on this exact question is the realism filter of Learning Defect Prediction from Unrealistic Data. It embeds real and synthetic functions with an off-the-shelf text embedding model and keeps the top fraction of synthetic examples nearest to real code. It then pretrains on that subset and fine-tunes on real data. We reimplement it on ETH Py150 with PyPIBugs as the real set, under its reported 25% subset. The reported numbers are the reference: on Draper vulnerability data, the 25% most realistic subset gained 1.42% accuracy and 7.17% F1 over a random 25% subset, and absolute accuracy on real-world bug prediction stayed around 60%. For fairness we also give this filter access to the target project, by using target embeddings as the realism anchor, so that it gets the same locality advantage. It receives the same number of synthetic training pairs, the same model, and the same hyperparameter search of 20 trials per condition.
The unsupervised domain-adaptation baseline follows the recipe of SDP-MTF. That method aligns source and target representations with moment matching and reports an average F1 of 0.7061 on 30 PROMISE project pairs. Because that number is for metric-plus-AST file-level prediction, it is not comparable to our function-level task. We therefore apply its adaptation step, matching the distribution of deep features on unlabeled target code, to the same CodeBERT detector. This baseline uses the same unlabeled target code our method uses, but it uses no execution.
Three ablations isolate the key ingredients. The first uses target-local mutants without any execution filter, which isolates the constraint. The second uses target-local behavior-changing mutants including those the tests catch, which isolates the escape requirement. The third uses escaped bugs simulated on the other projects rather than the target, which isolates locality. Beating a baseline means a gain in target F1 that holds when averaged over held-out projects, with the paired difference exceeding two standard errors across five seeds and across projects. Wall-clock generation cost is reported alongside.
Why not just…?
Why not just mutate the target project's code without running anything? This is the cheapest version and the real threat, because locality alone might deliver most of the gain. Equivalent mutants are label noise, and Alrasheed et al. show that noisy synthetic bugs can hurt. Still, nothing in the file measures how much of the harm comes from style mismatch versus non-behavior-changing bugs. The first ablation answers this directly. If unfiltered target-local mutants match the escaped set within noise, the execution constraint is unnecessary, and we will report that the contribution reduces to target-local augmentation.
Why not just filter an existing synthetic corpus by embedding similarity to the target? Embedding nearness measures surface resemblance, not whether a change is a real bug. Alrasheed et al.'s filtered models still sit around 60% accuracy on real bugs. The anchored-filter baseline tests this at matched data size and with the same access to the target project.
Why not just adapt to the target with unlabeled code, as transfer methods do? Feature alignment moves representations toward the target, but it supplies no target labels. Altinger et al. found that even within identically generated MISRA C code, normalization could not make faulty regions overlap across projects, and cross-project precision stayed at 0.17 to 0.24. The domain-adaptation baseline measures whether this ceiling also holds for function-level semantic detection.
Why not just rely on larger pretrained code models? CodeS shows that pretrained bimodal models resist token-level shift but still collapse under task shift. As an extra condition, we include a large code model prompted zero-shot on the target's real bugs. If it beats every fine-tuned condition, the cost of simulation is not worth paying at this scale.
Proposed Experiments
The main benchmark is PyPIBugs, about 4.5K real bug-fix functions from PyPI packages, with ETH Py150 as the external synthetic corpus. We split PyPIBugs by repository and keep the subset of repositories whose historical versions can be installed and whose test suites run. The number of such repositories is reported, because this filter itself shapes the population. Evaluation is leave-one-project-out, using accuracy and F1 for comparability with Alrasheed et al., plus AUC.
The decisive check is a three-way ordering. The target-local escaped-bug condition must outperform its unfiltered target-local ablation, and the unfiltered ablation must outperform the anchored realism filter. If the first gap vanishes, the physics analogy adds nothing beyond locality. If the escaped set beats the behavior-changing set that includes caught bugs, then the measurement model contributes something separate from the admissibility constraint.
A second check tests the mechanism rather than the score. The gain from the escaped set should be larger on projects where the cross-project baseline degrades most, and it should correlate with the project's escaped-bug fraction. A null result here would suggest that any improvement is regularization rather than distribution matching.
As a secondary probe of task shift, we train on CodeS's Python800-S split of CodeS, simulate escaped bugs in held-out problems, and ask whether simulation lifts accuracy on problems never seen in training. That is the regime where CodeS reports total failure.
References
- Learning Defect Prediction from Unrealistic Data
- Time series Foundation Models based on Physics-Informed Synthetic Histories for Cold-Start Photovoltaic Forecasting
- CodeS: Towards Code Model Generalization Under Distribution Shift
- ConTest: A Unit Test Completion Benchmark featuring Context
- A Mutation-Based Data Enhancement Approach for Software Vulnerability Detection
- Beyond respiratory models: a physics-enhanced synthetic data generation method for 2D-3D deformable registration
- Nuclei Segmentation of Fluorescence Microscopy Images Using Three Dimensional Convolutional Neural Networks
- SDP-MTF: A Composite Transfer Learning and Feature Fusion for Cross-Project Software Defect Prediction
- Novel Insights on Cross Project Fault Prediction Applied to Automotive Software