Automated Bug Generation in the era of Large Language Models

Ali Reza Ibrahimzada, Yang Chen, Ryan Rong, Reyhaneh JabbarvandarXiv 2023 · arXiv · OpenReview

TL;DR

The paper introduces BUGFARM, a novel method for generating complex, synthetic bugs designed to challenge modern machine learning models in software engineering. By leveraging LLMs and analyzing model attention, BUGFARM creates bugs that are both hard to detect and hard to repair, offering a more realistic evaluation of bug prediction and repair tools.

What Problem Does This Paper Address?

Evaluating the effectiveness of automated bug detection, localization, and repair techniques is crucial for improving software quality. However, existing methods often rely on simple bugs that are easily caught by testing or simple models. The challenge lies in creating "complex" bugs: those that are difficult to detect (e.g., requiring specific inputs or edge cases) and hard to fix (involving changes across multiple code locations). For machine learning models, this complexity is further nuanced. A bug needs to be similar enough to correct code in its representation to fool a prediction model, yet different enough in its actual behavior to be considered a bug. Existing bug generation techniques often fail to balance these competing requirements, leading to an overestimation of model performance.

Methods

BUGFARM tackles this challenge with a three-component framework:

  1. Method Extractor: This component parses the input project to identify individual methods and constructors.
  2. Attention Analyzer: This is the core innovation. It leverages the attention mechanisms of pre-trained code-language models (like CodeBERT, CodeT5) to identify code tokens and statements that contribute least to the model's representation of the code. The hypothesis is that modifying these "least attended" locations will result in bugs that are semantically similar to the original code, making them harder for models to detect.
  3. Bug Generator: This component uses a Large Language Model (LLM) to mutate code. Crucially, it prompts the LLM to introduce changes only in the least attended statements identified by the Attention Analyzer. This ensures the "hard-to-detect" property. The LLM is also instructed to make changes across multiple statements, addressing the "hard-to-repair" aspect. The generated bugs undergo validation for syntactic correctness and minimal impact on the model's attention.

Datasets

The paper uses 15 Java projects from the Defects4J v2.0 dataset for generating bugs. For training bug prediction models, they utilize real-world bug datasets (Defects4J, BugSwarm, RegMiner) and the synthetic bugs generated by BUGFARM, LEAM, and µBERT.

Key Insights and Ideas

  • Conflicting Objectives: The paper highlights the inherent tension between creating bugs that are hard to detect by ML models (requiring similar code representations) and hard to repair (requiring multiple code changes). BUGFARM's approach of targeting "least attended" locations is a clever way to reconcile these.
  • Attention as a Proxy for Detectability: The core idea that modifying "least attended" code segments makes bugs harder for attention-based models to detect is a significant insight. It suggests that model attention patterns can be directly leveraged to craft challenging test cases.
  • LLMs for Complex Mutations: Utilizing LLMs for bug generation allows for more creative and diverse mutations than traditional rule-based or template-based methods, especially for introducing multi-statement bugs.
  • BUGFARM Bugs are Unique and Challenging: The evaluation demonstrates that BUGFARM-generated bugs are less likely to overlap with those from other methods and are significantly harder for both bug prediction and program repair models to handle.

Key Results

  • Harder to Detect: BUGFARM bugs resulted in up to 40.53% higher False Negative Rates (FNR) and lower Accuracy, Precision, Recall, and F1 scores for bug prediction models compared to LEAM and µBERT bugs.
  • Harder to Repair: Automated program repair (APR) techniques successfully repaired only 28% of BUGFARM bugs, compared to 36% and 49% for LEAM and µBERT bugs, respectively. This is attributed to BUGFARM's tendency to create multi-statement bugs.
  • Efficiency: BUGFARM is efficient, taking approximately 9 seconds per method for bug generation (dominated by LLM prompting), and significantly faster in end-to-end mutant generation (74% faster than LEAM, 67% faster than µBERT).
  • Quality: BUGFARM generated a higher percentage of syntactically correct mutants (85%) compared to LEAM (59.4%) and µBERT (61.82%).
  • Human Study: A manual evaluation confirmed that BUGFARM mutants were more challenging for human annotators to label as buggy or equivalent.

Limitations and Future Work

  • Model Agnosticism vs. Specificity: While BUGFARM is designed to be language-agnostic, its Attention Analyzer relies on transformer-based models. The authors acknowledge that including closed-source models would be a future direction.
  • LLM Dependence: The performance of BUGFARM is tied to the capabilities of the LLM used for generation. The paper notes that using more advanced LLMs like GPT-4 could yield even better results.
  • Real-World Bug Equivalence: The paper focuses on synthetic bugs and does not claim to replace real-world bug datasets. However, they suggest that BUGFARM's bugs, generated with LLMs that have seen real-world data, might share similarities with human mistakes.
  • Future Work: The authors suggest exploring BUGFARM's applicability to other programming languages and incorporating more sophisticated LLMs. They also plan to investigate the impact of different threshold values for attention analysis and the generation of more diverse bugs.

Summary updated

Abstract

Bugs are essential in software engineering; many research studies in the past decades have been proposed to detect, localize, and repair bugs in software systems. Effectiveness evaluation of such techniques requires complex bugs, i.e., those that are hard to detect through testing and hard to repair through debugging. From the classic software engineering point of view, a hard-to-repair bug differs from the correct code in multiple locations, making it hard to localize and repair. Hard-to-detect bugs, on the other hand, manifest themselves under specific test inputs and reachability conditions. These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs, are mostly aligned; a bug generation technique can change multiple statements to be covered only under a specific set of inputs. However, these two objectives are conflicting for learning-based techniques: A bug should have a similar code representation to the correct code in the training data to challenge a bug prediction model to distinguish them. The hard-to-repair bug definition remains the same but with a caveat: the more a bug differs from the original code (at multiple locations), the more distant their representations are and easier to be detected. We propose BugFarm, to transform arbitrary code into multiple complex bugs. BugFarm leverages LLMs to mutate code in multiple locations (hard-to-repair). To ensure that multiple modifications do not notably change the code representation, BugFarm analyzes the attention of the underlying model and instructs LLMs to only change the least attended locations (hard-to-detect). Our comprehensive evaluation of 320k+ bugs from over 2.5M mutants generated by BugFarm and two alternative approaches demonstrates our superiority in generating bugs that are hard to detect by learning-based bug prediction approaches and hard to repair by SOTA learning-based program repair technique.
BibTeX
@misc{ibrahimzada2023automated,
  author = {Ali Reza Ibrahimzada and Yang Chen and Ryan Rong and Reyhaneh Jabbarvand},
  title = {{Automated Bug Generation in the era of Large Language Models}},
  year = {2023},
  eprint = {2310.02407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.02407},
}
Figure