On Comparing Mutation Testing Tools through Learning-based Mutant Selection

Milos Ojdanic, Ahmed Khanfir, Aayush Garg, Renzo Degiovanni, Mike Papadakis, Yves Le TraonAST 2023 · OpenReview

Abstract

Recently many mutation testing tools have been proposed that rely on bug-fix patterns and natural language models trained on large code corpus. As these tools operate fundamentally differently from the grammar-based traditional approaches, a question arises of how these tools compare in terms of 1) fault detection and 2) cost-effectiveness. Simultaneously, mutation testing research proposes mutant selection approaches based on machine learning to mitigate its application cost. This raises another question: How do the existing mutation testing tools compare when guided by mutant selection approaches? To answer these questions, we compare four existing tools – μBERT (uses pre-trained language model for fault seeding), IBIR (relies on inverted fix-patterns), DeepMutation (generates mutants by employing Neural Machine Translation) and PIT (applies standard grammar-based rules) in terms of fault detection capability and cost-effectiveness, in conjunction with standard and deep learning based mutant selection strategies. Our results show that IBIR has the highest fault detection capability among the four tools; however, it is not the most cost-effective when considering different selection strategies. On the other hand, μBERT having a relatively lower fault detection capability, is the most cost-effective among the four tools. Our results also indicate that comparing mutation testing tools when using deep learning-based mutant selection strategies can lead to different conclusions than the standard mutant selection. For instance, our results demonstrate that combining μBERT with deep learning-based mutant selection yields 12% higher fault detection than the considered tools.

Introduction: The Cost and Realism of Mutation Testing

Mutation testing is widely regarded as the "gold standard" for evaluating test suite quality. By intentionally injecting small errors (mutants) into source code, developers can determine if their test suites are capable of catching real-world bugs. However, the technique faces two significant hurdles: high computational and manual cost due to the generation of thousands of redundant mutants, and the "realism problem"—whether simple grammar-based changes (like swapping > for >=) accurately mimic complex human errors.

Recent advancements have introduced tools using Neural Machine Translation (NMT) and Large Language Models (LLMs) to seed more "natural" faults. This study investigates whether these AI-driven tools outperform traditional grammar-based approaches, particularly when paired with "smart" machine learning-based mutant selection.

Methodology

The researchers performed a head-to-head comparison of four fundamentally different mutation tools:

  • PIT: The industry standard utilizing manual grammar-based rules.
  • μBERT: Employs CodeBERT (an LLM) to replace code tokens with contextually likely alternatives.
  • IBIR: Utilizes "inverted fix-patterns" derived from historical real-world bug fixes.
  • DeepMutation: Uses NMT to "translate" healthy code into buggy variants.

These tools were evaluated using two selection strategies: Standard Selection (random) and Cerebro, a deep learning model designed to identify high-value "subsuming" mutants that provide the most information about test quality. The study utilized the Defects4J v2.0.0 benchmark, analyzing 509 real-world faults across 15 Java projects.

Key Results and Insights

The study revealed that the choice of selection strategy significantly alters the perceived effectiveness of mutation tools.

The Synergy of μBERT and Cerebro

A primary finding is the "synergy effect" between μBERT and Cerebro. Because μBERT mutants are "homogeneous"—typically replacing one token with another—the Cerebro selector can prioritize them more accurately than the structurally complex mutants generated by IBIR. When combined, μBERT and Cerebro yielded 12% higher fault detection than any other combination for the same level of effort.

Raw Power vs. Efficiency

While IBIR is the "brute force" leader, capable of finding approximately 90% of real-world faults, it produces a high volume of redundant mutants, making it less cost-effective. In contrast, μBERT (guided by Cerebro) required 82% fewer mutants to reach the same effectiveness level as the traditional PIT tool.

Comparison of fault detection capability vs. effort for different mutation tools and selection strategies.

LLM Realism and NMT Struggles

μBERT demonstrated its ability to mimic subtle, context-dependent human errors by finding 17 unique bugs that even the pattern-based IBIR missed. Conversely, DeepMutation struggled, often being "subsumed" by other tools, meaning its findings were redundant and required more effort to uncover.

Conclusion

The evaluation of mutation testing tools cannot be conducted in a vacuum; the selection strategy is a critical component of overall performance. While pattern-based tools like IBIR offer high raw fault detection, LLM-based tools like μBERT provide superior cost-effectiveness when paired with deep learning selectors. Future work should explore specialized classifiers for complex pattern-based tools and hybrid strategies that combine the strengths of both grammar-based and AI-driven mutation.

Summary updated

Abstract

Recently many mutation testing tools have been proposed that rely on bug-fix patterns and natural language models trained on large code corpus. As these tools operate fundamentally differently from the grammar-based traditional approaches, a question arises of how these tools compare in terms of 1) fault detection and 2) cost-effectiveness. Simultaneously, mutation testing research proposes mutant selection approaches based on machine learning to mitigate its application cost. This raises another question: How do the existing mutation testing tools compare when guided by mutant selection approaches? To answer these questions, we compare four existing tools – μBERT (uses pre-trained language model for fault seeding), IBIR (relies on inverted fix-patterns), DeepMutation (generates mutants by employing Neural Machine Translation) and PIT (applies standard grammar-based rules) in terms of fault detection capability and cost-effectiveness, in conjunction with standard and deep learning based mutant selection strategies. Our results show that IBIR has the highest fault detection capability among the four tools; however, it is not the most cost-effective when considering different selection strategies. On the other hand, μBERT having a relatively lower fault detection capability, is the most cost-effective among the four tools. Our results also indicate that comparing mutation testing tools when using deep learning-based mutant selection strategies can lead to different conclusions than the standard mutant selection. For instance, our results demonstrate that combining μBERT with deep learning-based mutant selection yields 12% higher fault detection than the considered tools.
BibTeX
@inproceedings{ojdanic2023comparing,
  author = {Milos Ojdanic and Ahmed Khanfir and Aayush Garg and Renzo Degiovanni and Mike Papadakis and Yves Le Traon},
  title = {{On Comparing Mutation Testing Tools through Learning-based Mutant Selection}},
  booktitle = {AST},
  year = {2023},
  url = {https://openreview.net/forum?id=7bWpV8Of6R},
}
Figure