Fuzzing-based Mutation Testing of C/C++ CPS

Jaekwon Lee, Fabrizio Pastore, Lionel C. Briand

TL;DR

This paper introduces MOTIF, a novel approach that uses fuzz testing to automate mutation testing for C/C++ software in cyber-physical systems (CPS). By leveraging fuzzing's ability to explore diverse program states, MOTIF effectively detects mutants (artificially injected faults) that traditional symbolic execution methods struggle with, especially in complex CPS environments.

What Problem Does This Paper Address?

Ensuring the quality and reliability of embedded software in safety-critical cyber-physical systems (CPS) is paramount. Mutation testing is a powerful technique for evaluating test suite quality by injecting faults (mutants) and checking if test cases can detect them. However, existing mutation testing techniques for C/C++ (the dominant languages in CPS) often rely on symbolic execution. Symbolic execution faces limitations with CPS, such as difficulties handling black-box components, complex environments, and floating-point operations. This leaves a gap in automated, effective mutation testing for these critical systems.

Methods

The core of MOTIF is its use of fuzz testing to generate test inputs for mutation testing. The process involves:

  1. Fuzzing Driver Generation: MOTIF analyzes the System Under Test (SUT) to generate a fuzzing driver that can execute both the original and mutated functions.
  2. Seed File Generation: It creates seed files for fuzzers based on input parameter types, aiming to cover basic cases.
  3. Compilation: The SUT, mutated functions, and fuzzing driver are compiled.
  4. Mutation Testing: A fuzzer (like AFL++) is used to generate inputs that exercise the fuzzing driver. MOTIF monitors for crashes or aborted executions, identifying inputs that kill mutants. It includes a post-processing step to mitigate false positives caused by non-determinism.
  5. SUT Testing: A test driver is generated to execute the original SUT with the mutant-killing inputs, allowing for verification and integration into test suites.

MOTIF-C++ extends this approach to handle C++'s object-oriented features like abstraction, encapsulation, inheritance, and polymorphism.

Datasets

The evaluation was performed on several real-world software components from operational satellite systems, including:

  • MLFS: Mathematical Library for Flight Software.
  • LIBU: A utility library from GomSpace.
  • ASN1lib: A serialization/deserialization library.
  • ESAIL: Control software for a micro-satellite.
  • S5: Ground software for processing satellite radiation data.

These subjects represent diverse features typical of CPS and were chosen for their industrial relevance.

Key Insights and Ideas

  • Fuzzing as a Mutation Testing Engine: The paper demonstrates that fuzzing, known for its effectiveness in exploring diverse program states, is a viable and powerful alternative to symbolic execution for mutation testing in CPS.
  • Bridging the Gap for CPS: MOTIF's approach directly addresses the limitations of symbolic execution in CPS, making mutation testing more accessible for these systems.
  • Configuration Matters: The choice of compiler (Clang), sanitizer (ASAN), and coverage metric (LAF) significantly impacts MOTIF's effectiveness, with a specific "Best" configuration identified.
  • Complementarity of Approaches: While fuzzing is effective, there's still a degree of complementarity with symbolic execution, suggesting hybrid approaches could offer further gains, though with increased complexity.
  • Cost-Effectiveness of Reuse: Reusing mutant-killing inputs can improve mutation scores with minimal additional testing time, especially for systems with a sufficient test budget.

Key Results

  • MOTIF can detect between 40% and 90% of mutants not detected by developers' test suites.
  • The "Best" configuration (Clang, ASAN, LAF) significantly outperforms a baseline (GCC) by 1.53 to 4.34 percentage points in mutant killing.
  • MOTIF outperforms a symbolic execution-based approach (SEMuP) by 13.18 to 50.33 percentage points on average for the tested subjects.
  • Hybrid fuzzing (MOTIF-Hybrid) shows a slight improvement in mutation score (up to 1.37 pp) and a reduction in kill time (up to 31.95%).
  • Reusing mutant-killing inputs improves the mutation score by an average of 1.40 pp across subjects.

Limitations and Future Work

  • Manual Tuning for C++: While MOTIF-C++ handles OOP features, manual tuning of test utility classes is sometimes required for parametric and subtype polymorphism. The authors suggest that improved static analysis could automate this.
  • Fuzzer Dependency: The effectiveness of MOTIF is tied to the performance of the underlying fuzzer (AFL++).
  • Hybrid Fuzzing Complexity: While beneficial, hybrid fuzzing (MOTIF-Hybrid) requires a more complex setup.
  • Future Work: The authors suggest exploring LLM-based fuzz driver generation, integrating more advanced fuzzing strategies, and investigating MOTIF's effectiveness for other types of mutants (e.g., cybersecurity-focused, AI-generated). They also plan to investigate the complementarity of MOTIF with other hybrid fuzzing solutions and address the mutation score plateau observed in some experiments.

Summary updated

BibTeX
@article{lee2026fuzzing,
  author = {Jaekwon Lee and Fabrizio Pastore and Lionel C. Briand},
  title = {{Fuzzing-based Mutation Testing of C/C++ CPS}},
  journal = {Empir. Softw. Eng.},
  year = {2026},
  doi = {10.1007/s10664-025-10700-7},
}

Loading related content...

Figure