Scaling Mutation Testing for Large Software Systems

29 concepts29 papers2 proposals

Mutation testing is often described as the "gold standard" for evaluating test suite quality. By injecting artificial faults—mutants—into a codebase, developers can measure how many of these bugs their tests actually catch. However, as Garg and Ojdanić observe, the technique faces a "mutant explosion" problem. In large-scale systems, generating and executing thousands of mutants against extensive test suites creates a massive computational bottleneck that makes the process impractical for modern continuous integration pipelines.

To bridge this "practicality gap," researchers have developed strategies to move away from exhaustive execution toward intelligent selection, redundancy removal, and family-based analysis.

Predicting Mutant Utility

The most direct way to reduce cost is to avoid running "useless" mutants. Not every artificial bug provides new information; many are either too easy to catch or impossible to trigger.

Machine learning has emerged as a primary tool for this selection process. For instance, Titcheu Chekam et al. introduced FaRM (Fault Revealing Mutants), which uses static code features to predict which mutants are likely to reveal real-world faults. By focusing only on these high-value mutants, the authors found they could uncover more bugs while significantly reducing computational waste.

More recent approaches leverage deep learning to bypass execution entirely. Alon et al. and the authors of MutationBERT use transformer-based models to predict whether a test will catch a specific mutant without actually running the code. Similarly, Garg and Ojdanić developed Cerebro, a deep learning approach that statically predicts "subsuming" mutants—those that, if caught, imply that other mutants are also caught—reducing execution costs by up to 90%.

Eliminating Redundancy and Equivalence

A significant portion of the computational burden in mutation testing comes from "equivalent mutants"—changes that alter the source code but do not change the program's observable behavior. Because no test can ever "kill" an equivalent mutant, running them is a pure waste of resources.

To combat this, Basile et al. proposed formal "Commandments" for the static detection of equivalent mutants in real-time systems. By filtering these out before execution, engineers can avoid expensive reachability analysis for faults that are logically invisible. In the domain of safety-critical timed systems, Cuartas et al. introduced MUPPAAL, which uses timed bisimulation algorithms to identify and remove over 99% of duplicate mutants, ensuring that every executed test run contributes new information to the quality assessment.

Family-Based and Contextual Analysis

Rather than treating every mutant as an independent entity, researchers have proposed treating them as a collective. Devroey et al. suggest a "variability perspective," where mutants are modeled as a family. This allows a test suite to be evaluated against an entire set of variations simultaneously, rather than sequentially. Basile et al. note that transitioning to this family-based checking remains a key frontier for scaling mutation testing to large-scale architectures.

Efficiency can also be gained by narrowing the scope of testing to what has actually changed in the software. In rapidly evolving systems, Ojdanic et al. argue for commit-relevant mutants. These are mutants that specifically interact with the "delta" of a code change. The authors demonstrated that these mutants are 30% more effective at catching bugs related to new commits than traditional, codebase-wide mutation strategies. By focusing on how new code interacts with existing logic, "commit-aware" mutation testing makes the technique viable for rapid development cycles (Ojdanic et al.).

Domain-Specific Optimizations

Finally, some researchers achieve scalability by tailoring the mutation process to specific types of software.

  • Scientific Applications: Schaad et al. developed FuzzyFlow, which uses dataflow analysis to extract minimal "cutouts" from massive applications. Testing these isolated fragments is up to 528x faster than whole-program testing.
  • Graph Neural Networks: Yang et al. introduced GraphRank to prioritize test inputs, avoiding the prohibitive cost of retraining model variants for every mutation.
  • Cyber-Physical Systems: The MASS pipeline, proposed by Cornejo et al., integrates sampling and optimization to make mutation analysis practical for complex embedded systems in the space domain.

While Sánchez et al. report that computational cost remains the primary barrier to professional adoption, these advancements in selection, pruning, and contextual awareness are transforming mutation testing from an academic ideal into a scalable industrial tool.

Key Foundational Work

Computing Mutation Coverage in Interpolation-Based Model Checking (2012) — contributed 1 concept elements to this direction.
From genetic to bacteriological algorithms for mutation-based testing (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computational_cost_of_mutation_testing_ee942695,
  title = {computational cost of mutation testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-mutation-testing/rdr_ee9426957bb04ba385c141b00d181760},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_computational_cost_of_mutation_testing_ee942695_v1,
  title = {computational cost of mutation testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-mutation-testing/rdr_ee9426957bb04ba385c141b00d181760/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake