Abstract
As Large Language Models (LLMs) become increasingly integrated into software engineering workflows, particularly for Fault Localization (FL) and Program Repair (PR), the integrity of evaluation benchmarks has come under scrutiny. Existing benchmarks often predate the training data cutoffs of modern LLMs, leading to "data leakage" where models may simply recall memorized solutions. This paper introduces ConDefects, a dataset of approximately 2,900 real-world software bugs and patches sourced from competitive programming submissions created after September 2021. By ensuring the data was generated after the training cutoff of major models like GPT-3.5 and GPT-4, ConDefects provides a more rigorous and valid assessment of LLM capabilities in software maintenance tasks.
1. Introduction
Large Language Models have demonstrated remarkable potential in automating complex software engineering tasks. However, the research community faces a significant "validity threat": data leakage. Most popular benchmarks, such as Defects4J or ManyBugs, have been publicly available on platforms like GitHub for years. Because LLMs are trained on massive web crawls, there is a high probability that these benchmarks were included in their training sets. Evaluating an LLM on such data is akin to providing a student with an exam they have already memorized, leading to inflated and misleading performance metrics.
To address this, we present ConDefects, a dataset specifically designed to be "unseen" by contemporary LLMs, enabling a true zero-shot evaluation of their fault localization and repair abilities.
2. Methodology
The ConDefects dataset was curated by mining the AtCoder platform, a prominent competitive programming site. The authors implemented a multi-stage pipeline to ensure data quality and relevance:
2.1 Temporal Filtering
To mitigate data leakage, the authors selected coding tasks published between October 2021 and September 2023. This specific window falls after the knowledge cutoff for many foundational LLMs, ensuring that the bugs and their corresponding fixes were not part of the models' initial training corpora.
2.2 The "One-Line" Heuristic
To maintain a clean and well-defined dataset, the authors focused on single-line faults. They identified user submission histories where a "Wrong Answer" (WA) was immediately followed by an "Accepted" (AC) submission. By isolating pairs where the fix required only a single-line modification, the dataset provides clear, unambiguous ground truth for both localization and repair.
2.3 Multi-Language and Metadata Support
Recognizing the diversity of the software ecosystem, ConDefects includes samples in both Java and Python. Each entry is enriched with metadata, including the exact faulty line number, the corrected code, task difficulty levels (based on AtCoder ratings), and precise submission timestamps.
3. Dataset Characteristics
The final ConDefects dataset comprises 2,879 total bug-patch pairs derived from 645 distinct coding tasks. The distribution is as follows:
- Java: 1,254 faulty programs (Average 180.92 LOC, 16 functions).
- Python: 1,625 faulty programs (Average 40.91 LOC).
Unlike synthetic datasets that rely on mutation testing (randomly injecting errors), ConDefects captures real human errors made by developers under competitive pressure, reflecting genuine logical and syntax-related mistakes.
4. Evaluation and Key Results
To validate the difficulty and necessity of ConDefects, the authors established baselines using standard Spectrum-Based Fault Localization (SBFL) algorithms, specifically the Ochiai formula.
4.1 Performance Disparity
A comparison between ConDefects and the classic Defects4J benchmark revealed a significant performance gap. On the Java subset of ConDefects, the Ochiai formula achieved a Top-1 accuracy of only 5.72%, compared to approximately 14.7% on Defects4J. This suggests that ConDefects is not only "unseen" but also inherently more challenging, providing a more realistic testbed for modern algorithms.
4.2 Baseline Metrics
The localization performance on the Java subset was recorded as follows:
- Top-1 Accuracy: 5.72%
- Top-3 Accuracy: 20.34%
- Top-5 Accuracy: 36.86%
These results underscore the importance of using fresh, non-leaked data to evaluate the next generation of AI-driven software tools.
5. Discussion and Future Work
While ConDefects addresses the critical issue of data leakage, it has certain limitations. Currently, the dataset is restricted to single-line faults, whereas real-world industrial bugs often span multiple files or complex architectural layers. Furthermore, competitive programming code, while logically dense, may not fully mirror the complexities of enterprise-scale software systems.
Future iterations of ConDefects will aim to include multi-line fixes and a broader range of software domains. As LLMs continue to evolve and ingest newer data, the authors intend to provide continuous updates to the dataset, ensuring it remains a "future-proof" benchmark for the software engineering community.
Availability: The ConDefects dataset and its accompanying Python toolkit are publicly available for research purposes at https://github.com/appmlk/ConDefects.
Summary updated