A Comprehensive Study on Large Language Models for Mutation Testing

Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

TL;DR

This paper presents the first large-scale comprehensive empirical study evaluating Large Language Models (LLMs) for mutation testing. By comparing leading open- and closed-source LLMs against traditional rule-based and deep-learning baselines, the authors show that LLMs generate significantly more diverse and realistic mutants, yielding up to a 1.75× improvement in real bug detection. However, this superior effectiveness comes with a "validity tax"—namely, higher rates of non-compilability, code duplication, and computational overhead.

What Problem Does This Paper Address?

Mutation testing is a powerful software engineering technique used to evaluate the quality of a test suite. By intentionally introducing small, artificial errors (mutants) into a codebase, developers can see if their existing tests are robust enough to catch them.

Historically, mutation testing tools have relied on rigid, predefined syntactic rules (e.g., swapping a + for a -). While reliable, these rule-based mutants often fail to mimic the subtle, complex, and context-dependent nature of real-world human programming errors. Tech giants like Meta and Google have recently begun deploying LLMs to generate more realistic mutants, but the research community lacked a rigorous, large-scale baseline to evaluate the true effectiveness, validity, and efficiency trade-offs of LLM-based mutation testing.

Methods

The authors conducted a massive empirical study tracking over 701,400 mutants. They evaluated two state-of-the-art LLM mutant generation frameworks—BugFarm and LLMorpheus—alongside their own newly designed prompt template, LLMut.

As shown in the study's approach overview, the evaluation spans a wide variety of tools:

Table 1. Studied Mutant Generation Approaches

The LLMut prompt template utilizes a few-shot learning strategy, providing the model with context from the surrounding method and structured JSON requirements to enforce clean formatting:

Fig. 1. Our New Prompt Template for Mutant Generation

The few-shot examples were carefully sampled from an independent dataset to teach the LLM how real bugs look without polluting the evaluation data:

Table 2. The Default Few-Shot Examples of LLMut from QuixBugs

The experiments evaluated seven distinct open- and closed-source LLMs (including GPT-4o, GPT-3.5-Turbo, and DeepSeek-V3-671b) against four traditional baselines: two rule-based tools (PIT and Major) and two smaller, custom-trained deep learning models (LEAM and $\mu$BERT).

Datasets

The mutant generation approaches were evaluated on 851 real-world bugs across two major Java benchmarks:

  • Defects4J 2.0: 605 bugs from 12 widely used open-source Java projects.
  • ConDefects: 246 recent bugs (from March to June 2024). This dataset was specifically included to mitigate the risk of data leakage, ensuring the newer LLMs hadn't already seen these bugs during their pre-training phases.

A breakdown of the specific projects and time spans can be seen below:

Table 3. Real-world Bugs Used in Our Study

Key Insights and Ideas

  • Semantic Proximity to Real Faults: LLM-generated mutants are behaviorally much closer to real human errors. Instead of simple single-token changes, LLMs frequently perform multi-node Abstract Syntax Tree (AST) transformations, altering logic in a context-aware manner (e.g., understanding that a rightBlock variable implies the existence of a leftBlock and mutating it accordingly).
  • LLM Creativity: Surprisingly, LLMs proved highly creative. They introduced up to 49 unique new AST node types during mutation, vastly outperforming traditional rule-based tools (which only introduced 2) and far exceeding the diversity present in the few-shot examples provided in the prompt.
  • The "Validity Tax": The impressive leap in bug detection capability comes at a cost. LLMs lack the strict syntactic guarantees of rule-based engines, frequently generating mutants that fail to compile due to "hallucinated" methods or broken code structures.

Key Results

  • Superior Bug Detection: LLM-based approaches achieved a real bug detection rate of 76.47% compared to just 44.15% for traditional rule-based techniques—an absolute gain of 32.99 percentage points. The best-performing configuration, LLMut powered by DeepSeek-V3-671b, reached a staggering 91.1% fault detection rate.
  • The Downside of LLMs: Compared to rule-based approaches, LLM-generated mutants suffered from worse non-compilability (by 32.73 percentage points), higher duplication rates (by 9.38 percentage points), and slightly higher equivalent mutant rates (by 3.64 percentage points).
  • Efficiency Bottleneck: Traditional tools like PIT and Major are incredibly fast, generating a mutant in 0.02 to 0.08 seconds. LLMs are orders of magnitude slower, requiring anywhere from 1.31 to 9.06 seconds per mutant, alongside significant token costs.

Limitations and Future Work

The study focused exclusively on the Java programming language, meaning the findings might vary for dynamically typed languages or different runtime environments. Additionally, the high rate of non-compilable mutants represents a major waste of computational resources.

The authors suggest that future research should focus on integrating LLM mutant generation with static analysis tools or automated program repair techniques. By filtering out or fixing syntactically invalid mutants before they are executed, researchers can preserve the incredible fault-revealing power of LLMs while drastically reducing the computational overhead.

Summary updated

Abstract

Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we conduct a comprehensive empirical study evaluating BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches), alongside seven LLMs using our newly designed prompt, including both leading open- and closed-source models, on 851 real bugs from two Java real-world bug benchmarks. Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection. That is, 87.98% (for LLMs) vs. 41.64% (for rule-based); an increase of 46.34 percentage points. Nevertheless, our results also reveal that these impressive results for improved effectiveness come at a cost: the LLM-generated mutants have worse non-compilability, duplication, and equivalent mutant rates by 26.60, 10.14, and 3.51 percentage points, respectively. These findings are immediately actionable for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.
BibTeX
@misc{wang2024comprehensive,
  author = {Bo Wang and Mingda Chen and Ming Deng and Youfang Lin and Mark Harman and Mike Papadakis and Jie M. Zhang},
  title = {{A Comprehensive Study on Large Language Models for Mutation Testing}},
  year = {2024},
  eprint = {2406.09843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.09843},
}
Figure