Fuzzing-based mutation testing of C/C++ software in cyber-physical systems

Jaekwon Lee, Fabrizio Pastore, Lionel C. Briand

TL;DR

This paper introduces MOTIF, a novel approach that uses fuzzing to automate mutation testing for C/C++ software in cyber-physical systems (CPS). By leveraging fuzzing's ability to explore diverse program states, MOTIF effectively detects mutants that traditional symbolic execution methods struggle with, especially in complex CPS environments.

What Problem Does This Paper Address?

Ensuring the quality of embedded software in safety-critical cyber-physical systems (CPS) is paramount. Mutation testing is a powerful technique for evaluating test suite quality by injecting faults (mutants) and checking if test cases can detect them. However, existing mutation testing techniques for C/C++ (common in CPS) often rely on symbolic execution, which has limitations: it struggles with black-box components, requires extensive environment modeling, and can be computationally expensive. This paper addresses the need for an automated, practical mutation testing approach for C/C++ CPS software that overcomes these limitations.

Methods

The core of the paper is the MOTIF (Mutation TestIng with Fuzzing) approach. It works by:

  • Generating Fuzzing Drivers: For each mutated function, MOTIF creates a driver that uses fuzzing to generate test inputs.
  • Leveraging SOTA Fuzzers: MOTIF integrates with existing grey-box fuzzers (like AFL++) that use bucket-based coverage strategies to explore program states effectively.
  • Comparing Original and Mutated Functions: The fuzzing driver executes both the original and mutated functions with the same inputs and compares their outputs. Divergent outputs indicate a killed mutant.
  • Handling Nondeterminism: MOTIF includes a post-processing step to re-execute inputs that appear to kill mutants using a "false positive driver" to filter out cases where differences are due to nondeterministic behavior rather than actual faults.
  • C++ Extensions: MOTIF has been extended to handle C++'s object-oriented features like inheritance, abstraction, encapsulation, and polymorphism, often by generating test utility classes and copying mutated classes.

Datasets

The paper evaluates MOTIF on several real-world software components from operational satellite systems:

  • LIBU: A utility library from GomSpace.
  • ASN1lib: A serialization/deserialization library.
  • MLFS: A mathematical library for flight software.
  • ESAIL: A subset of control software for a micro-satellite.
  • S5: Ground software for processing radiation data from the Sentinel-5 mission.

These subjects are implemented in C and C++ and are representative of CPS software.

Key Insights and Ideas

  • Fuzzing as a Mutation Testing Enabler: The paper demonstrates that fuzzing, known for its effectiveness in exploring program states, can be a powerful alternative to symbolic execution for mutation testing, especially in complex CPS.
  • Complementarity of Fuzzing and Symbolic Execution: While fuzzing is generally more effective for MOTIF, there's a degree of complementarity, suggesting that hybrid approaches (combining fuzzing and symbolic execution) can offer marginal improvements.
  • Configuration Matters: The choice of compiler (Clang), sanitizer (ASAN), and coverage metric (LAF) significantly impacts MOTIF's performance, with a specific "Best" configuration identified.
  • Seed Inputs vs. Fuzzed Inputs: While seed inputs help kill some mutants, the majority are killed by inputs generated through the fuzzing process, highlighting the importance of fuzzing's exploration capabilities.
  • Cost-Effectiveness of Input Reuse: Reusing mutant-killing inputs to test other mutants can improve mutation scores with minimal additional time cost, especially when test budgets are limited.

Key Results

  • Mutant Detection: MOTIF, with its "Best" configuration (Clang, ASAN, LAF), can detect between 40% and 83% of mutants across the studied subjects.
  • Comparison with Symbolic Execution (SEMuP): MOTIF outperforms SEMuP (a symbolic execution-based tool) by a significant margin, killing 13.18 to 50.33 percentage points more mutants on average.
  • Hybrid Fuzzing: MOTIF-Hybrid (combining fuzzing with SymCC for symbolic execution) shows a slight increase in mutation score (up to 1.37 pp) and a reduction in kill time for some subjects.
  • Input Reuse: Reusing mutant-killing inputs improves the mutation score by an average of 1.40 pp across subjects, with minimal impact on execution time.

Limitations and Future Work

  • Manual Tuning for C++: While MOTIF-C++ handles OOP features, it still requires manual tuning of test utility classes for parametric and subtype polymorphism. The authors suggest that improved static analysis could automate this.
  • Fuzzer Dependency: The effectiveness of MOTIF is tied to the performance of the underlying fuzzer (AFL++).
  • Limited Scope of Mutation Operators: The study uses mutation operators commonly implemented in tools like MASS, and future work could explore other types of mutants (e.g., AI-generated, cybersecurity-focused).
  • Hybrid Fuzzing Complexity: While MOTIF-Hybrid offers benefits, its more complex setup might limit practical adoption compared to MOTIF.
  • Investigating Other Hybrid Fuzzers: The study acknowledges that some hybrid fuzzing tools were not applicable to all subjects, suggesting future work to explore these.
  • Addressing Mutation Score Plateaus: Future work could investigate solutions for mutation score plateaus, potentially due to equivalent mutants.

Summary updated

BibTeX
@article{lee2026fuzzing,
  author = {Jaekwon Lee and Fabrizio Pastore and Lionel C. Briand},
  title = {{Fuzzing-based mutation testing of C/C++ software in cyber-physical systems}},
  journal = {Empir. Softw. Eng.},
  year = {2026},
  doi = {10.1007/s10664-025-10700-7},
}
Figure