Measuring the Quality of B Abstract Machines with ISO/IEC 25010

Chenghao Cai, Jing Sun, Gillian Dobbie

Abstract

Formal verification methods typically focus on binary correctness—whether a model satisfies its invariants or not. However, a technically correct model may still suffer from poor design, lack of maintainability, or incomplete functional coverage. This paper introduces a framework for quantifying the quality of B Abstract Machines by adapting the ISO/IEC 25010 software quality standard. By defining a suite of semantic metrics, the authors enable developers to evaluate formal designs as multi-dimensional quality vectors, facilitating better decision-making during the early stages of system development.

Introduction

In the realm of formal methods, model checkers are primarily used to identify counterexamples that violate system properties. While essential for debugging, this "pass/fail" paradigm provides little insight into the overall quality of a design. A model might be free of invariant violations yet remain inefficient, difficult to understand, or fragile under modification.

Existing metrics for formal models often rely on simplistic measures like lines of code or are borrowed directly from source code analysis, failing to capture the semantic richness of a formal specification. There is a clear need for a unified, standard-based approach to measure whether a formal design is "good" before it proceeds to implementation. This research addresses this gap by mapping the ISO/IEC 25010 international standard to the B-method.

Methodology

The proposed framework maps six characteristics from ISO/IEC 25010—Functional Suitability, Reliability, Performance Efficiency, Usability, Security, and Maintainability—to specific metrics within the B-method environment.

State-Transition Metrics

The authors distinguish between "Total" and "Partial" metrics for completeness and correctness. Total metrics are binary indicators of requirement satisfaction. In contrast, Partial metrics utilize a similarity function to quantify how closely a model's state transitions align with desired requirements. This allows the framework to recognize "near misses" rather than treating all errors as equally catastrophic.

Fault Injection and Modularity

To evaluate Reliability and Maintainability, the framework employs fault injection. By randomly introducing "mutations" (adding or removing state transitions), the authors observe the impact on system behavior and invariant satisfiability. Furthermore, modularity analysis measures the independence of operations, quantifying how changes in one part of the machine propagate through the system.

Tool Integration

These metrics are implemented in a B model quality evaluator that leverages the ProB model checker. The tool explores the state space of a machine to compute the quality vector automatically.

Experimental Evaluation

The framework was evaluated using two distinct datasets: pedagogical clock models and a real-world cruise controller.

Clock Models (CM1–CM6)

Six small B models of a digital clock were used to demonstrate how specific design flaws reflect in quality scores. These models range from simple implementations to those with intentional errors, such as off-by-one errors in minute increments.

B Abstract Machines CM5 and CM6 representing different clock model implementations.

As shown in Figure 1, different modeling choices (such as the use of SELECT vs PRE clauses) can be analyzed to see how they affect the machine's functional suitability and clarity.

Volvo Cruise Controller (VCC)

The authors compared an original model (VCC0) from the ProB Public Example Repository against a degraded version (VCC1). In VCC1, state transitions were intentionally modified to test the sensitivity of the metrics. The results were telling:

  • Functional Suitability: VCC0 scored 1.000, while VCC1 dropped to 0.913 in Total Functional Completeness (TFComp).
  • Structural Integrity: The modifications in VCC1 caused a significant drop in Modularity (from 0.913 to 0.746), illustrating how functional changes can degrade the underlying architecture.
  • Performance: The evaluator recorded higher CPU time and peak memory usage for VCC1, indicating that the degraded design was more computationally expensive to verify.

Key Insights

This research shifts the focus from binary correctness to a nuanced view of quality as a vector. By introducing partial metrics, the framework provides a more granular understanding of model behavior. Additionally, the concept of "Accountability"—measuring how uniquely a state can be traced back to its origin—provides a new way to assess the diagnosability of formal designs.

Conclusion and Future Work

While the current framework focuses on semantic state-transition analysis, future iterations aim to incorporate syntactic analysis to better measure "Learnability." Furthermore, the authors plan to extend the metrics to handle temporal logics (LTL/CTL) for complex behavioral requirements. Ultimately, these quality scores could serve as fitness functions for the automatic repair of formal models, using AI to iteratively improve design quality.

Summary updated

BibTeX
@inproceedings{cai2020measuring,
  author = {Chenghao Cai and Jing Sun and Gillian Dobbie},
  title = {{Measuring the Quality of B Abstract Machines with ISO/IEC 25010}},
  booktitle = {Tase},
  year = {2020},
  doi = {10.1109/tase49443.2020.00031},
}
Figure