Abstract: Mutation testing is an established software quality assurance technique for the assessment of test suites. While it is well-suited to estimate the general fault-revealing capability of a test suite, it is not practical and informative when the software under test must be validated against specific requirements. This is often the case for embedded software, where the software is typically validated against rigorously-specified safety properties. In such a scenario (i) a mutant is relevant only if it can impact the satisfaction of the tested properties, and (ii) a mutant is meaningfully-killed with respect to a property only if it causes the violation of that property. To address these limitations of mutation testing, we introduce property-based mutation testing, a method for assessing the capability of a test suite to exercise the software with respect to a given property. We evaluate our property-based mutation testing framework on Simulink models of safety-critical Cyber-Physical Systems (CPS) from the automotive and avionic domains and demonstrate how property-based mutation testing is more informative than regular mutation testing. These results open new perspectives in both mutation testing and test case generation of CPS.
Introduction
In the development of safety-critical Cyber-Physical Systems (CPS)—such as autonomous vehicles or medical devices—ensuring that software adheres to strict safety requirements is paramount. Traditional Mutation Testing (MT) is a common technique used to evaluate the quality of test suites by injecting small faults (mutants) into the code. If a test suite can distinguish the mutated version from the original, the mutant is considered "killed," signaling a high-quality test suite.
However, in the context of data-flow models like Simulink, traditional MT often provides a false sense of security. Because almost any change in these models propagates to the output, mutants are easily "killed" by minor numerical differences that have no impact on the system's actual safety requirements. This paper introduces Property-Based Mutation Testing (PBMT), a framework that redefines mutant detection based on the violation of formal safety properties.
The Limitations of Traditional Mutation Testing
The primary issue with standard MT in CPS is its extreme sensitivity. A test case might be flagged as successful simply because a mutant causes a 1% change in a signal, even if that signal remains well within safe operating limits. This leads to an inflated mutation score that does not reflect the test suite's ability to detect safety-critical failures.
The Property-Based Mutation Testing Framework
PBMT shifts the focus from general output-checking to property-specific validation. The framework introduces several key concepts:
- $\phi$-Killed Mutants: A mutant is only considered killed if a test case that passes on the original program specifically triggers a violation of a formal safety property $\phi$ (expressed in Signal Temporal Logic, or STL) in the mutated program.
- $\phi$-Trivial Mutants: These are mutants that change the output but are logically or physically incapable of ever violating property $\phi$. Identifying these helps engineers focus on relevant faults.
- Search-Based Test Generation (SBTG): To find inputs that can kill "hard" mutants, the authors utilize the Blood Coagulation Algorithm (BCA). This optimization strategy searches for test inputs that maximize the difference between original and mutated signals in a way that specifically pushes the system toward a property violation.
Experimental Evaluation
The framework was evaluated using two industrial-strength Simulink benchmarks:
- Automatic Transmission Controller System (ATCS): Focuses on keeping engine and vehicle speeds below safety thresholds.
- Aircraft Elevator Control System (AECS): Focuses on the time required for an elevator to stabilize after a command.
Mutants were generated using the Fault Injection and Mutation (FIM) tool, applying operators such as "Stuck-at" and "Relational Operator Replacement."
Results and Discussion
The quantitative results reveal a massive "False Security Gap." In the ATCS model, a standard Adaptive Random Testing suite achieved a 100% mutation score under traditional MT. However, when evaluated using PBMT, the score plummeted to just 53.19%. This indicates that nearly half of the mutants "killed" by the suite did not actually represent a detected safety violation.
Furthermore, the study showed that even specialized Falsification Testing (FT) suites—designed specifically to break properties—only achieved PBMT scores between 42% and 57%. The authors' Search-Based strategy proved that many mutants remained "live" not because they were equivalent to the original, but because standard testing strategies failed to find the highly specific inputs required to trigger a violation.
Conclusion
Property-Based Mutation Testing provides a more rigorous and informative metric for safety-critical systems than traditional methods. By focusing on property violations rather than simple output deviations, PBMT helps engineers identify genuine gaps in their testing strategies. Future work will look to expand this framework to Higher Order Mutants (multiple interacting faults) and explore ways to reduce the computational cost associated with search-based test generation.
Summary updated