Introduction
Autonomous systems, ranging from self-driving vehicles to industrial robotics, must navigate environments characterized by inherent stochasticity and uncertainty. Ensuring the safety of these systems requires real-time monitoring capable of predicting failure events—such as collisions or mechanical overflows—within a near-term horizon. However, a significant technical gap exists between rigorous design-time verification and real-time execution.
Exhaustive reachability analysis, while formally sound, is computationally prohibitive for online use. Conversely, simplified data-driven models often lack formal guarantees and risk "overconfidence," potentially reporting a system as safe when it is at high risk. This paper introduces a scalable framework for "conservative" safety monitors: systems that are fast enough for runtime application but are theoretically guaranteed to underestimate safety rather than overestimate it, providing a reliable early warning system.
Methodology
The proposed framework utilizes a two-phase approach that decouples heavy computational verification from high-frequency runtime monitoring.
1. Design-Time Abstraction and Verification
To handle the complexity of continuous stochastic systems, the authors employ Interval Abstraction. The state space is partitioned into hyperrectangles, which are then used to construct a Probabilistic Automaton (PA). This abstraction ensures that if any point within a region can transition to another, the PA includes that transition, thereby overapproximating the real system's possible behaviors.
Using the PRISM probabilistic model checker, the authors calculate the minimum probability of the system remaining safe over a fixed time horizon ($T$) for every possible state in the abstract model. These results are compiled into a lookup table that maps abstract states and control actions to safety probabilities.
2. Runtime Monitoring
During operation, the system utilizes a Bayesian filter to maintain a state estimate. The authors evaluate two primary monitoring strategies:
- Point Estimate Monitoring: This method uses the mean of the state estimate to retrieve a single safety value from the lookup table.
- State Distribution Monitoring: This more robust method considers the entire probability distribution of the state estimate, computing a weighted sum of safety probabilities across the lookup table.
The core theoretical contribution is a proof demonstrating that if the abstract model is conservative and the state estimator is "well-calibrated" (i.e., its predicted probabilities align with observed frequencies), the resulting runtime safety estimates will remain conservative.
Experimental Evaluation
The framework was evaluated using a simulated water tank system involving two tanks with draining dynamics and a central controller. The model complexity was significant, involving 101 discrete water levels per tank and five control configurations, resulting in over 51,000 initial states for the model checker.
The monitors were tested across 500 trials of 50 time steps each to assess accuracy and calibration.
Key Results and Insights
The Virtue of Pessimism
In safety-critical engineering, underestimating safety is a protective feature. A conservative monitor ensures that if a safety probability of 80% is reported, the true probability is at least 80%. The authors' empirical tests confirmed that the monitors consistently stayed on the "safe" side of the calibration curve, validating the theoretical claims of non-overconfidence.
Superiority of Distribution-Based Monitoring
A key finding was that utilizing the full distribution from a state estimator (such as a particle filter) significantly outperforms point-estimate methods. The distribution-based monitor captures the "spread" of uncertainty, leading to superior calibration and reliability.
Performance Metrics
- Calibration: The State Distribution monitor achieved an Expected Calibration Error (ECE) of 0.00252, indicating exceptional alignment between predicted and actual safety outcomes.
- Accuracy: Using Receiver Operating Characteristic (ROC) curves, the distribution-based monitor demonstrated an Area Under the Curve (AUC) of 0.867, performing nearly as well as a "True State" monitor with perfect system knowledge.
Conclusion and Future Work
By moving the intensive model-checking process to the design phase, the authors have enabled a lightning-fast table-lookup monitor suitable for high-frequency autonomous systems. While the current approach excels in controlled simulations like the water tank system, future work is required to address the "curse of dimensionality" in higher-dimensional systems, such as drone swarms. Additionally, the authors aim to explore automated methods for learning discrete abstractions directly from data while maintaining formal safety bounds.
Summary updated