TL;DR
This paper introduces a framework for learning "runtime monitors" that ensure autonomous systems, particularly those using black-box machine learning (ML) components, stay within their safe operational design domain (ODD). By learning from simulations and using counterexample-guided refinement, these monitors predict when a system is about to exit its safe operating conditions, enabling a switch to a fallback safety controller.
What Problem Does This Paper Address?
Autonomous systems, like self-driving cars, increasingly rely on complex ML models for critical tasks like perception and control. However, these ML models can be brittle, failing unexpectedly when encountering situations outside their training data distribution (the "distribution gap"). This brittleness poses a significant safety risk. While the concept of an Operational Design Domain (ODD) aims to define safe operating conditions, simply defining an ODD isn't enough; it needs to be actively monitored at runtime. The challenge is to create monitors for ODDs of "black-box" ML components, where we don't have direct access to their internal workings or training data, and where the ODD itself might not be directly observable.
Methods
The authors propose a simulation-based framework that integrates with VERIFAI, an open-source toolkit for formal system analysis. The core idea is to:
- Simulation-based Analysis: Use VERIFAI and SCENIC (a probabilistic programming language) to generate diverse simulation scenarios of the autonomous system in its environment.
- Data Generation and Learning: Collect "traces" of observable features from these simulations. These traces are then processed (filtered, transformed) to create a training dataset. A learner then trains a "monitor" based on this data. The key is that the ODD is learned as a "monitorable" property, meaning it can be checked using observable features.
- Counterexample-Guided Refinement: The framework uses a conformance tester that actively searches for "counterexamples" – situations where the learned monitor incorrectly predicts safety or danger. These counterexamples are fed back to the data generation and learning components to refine the monitor iteratively.
- Monitor Types: They investigate two types of monitors:
- Stateless monitors: Make predictions based on the current observation of features.
- State-based monitors: Make predictions based on a sequence of recent observations, allowing them to capture temporal dynamics.
Datasets
The paper uses simulations generated by the CARLA simulator [5] and modeled using SCENIC [8]. The training and testing data are derived from these simulations, not from pre-existing, publicly released datasets for this specific ODD monitoring task. The simulations cover various weather conditions, road types, initial positions, and sun angles.
Key Insights and Ideas
- Monitorability is Crucial: Simply defining an ODD is insufficient for safety-critical systems. The ODD must be expressible in terms of observable features and implementable as a runtime monitor.
- State-based Monitors Outperform Stateless Ones: Capturing the temporal evolution of system states (using sequences of observations) significantly improves the ability of monitors to detect potential ODD violations, leading to fewer false negatives.
- Counterexample-Guided Learning is Effective: Iteratively refining the monitor based on counterexamples found during conformance testing is key to learning accurate and robust ODD monitors.
- Balancing Complexity and Delay: While state-based monitors offer better safety, they introduce computational delay. The paper highlights the need to balance the length of input sequences for the monitor with the acceptable latency.
Key Results
- The learned monitors (both stateless and state-based) significantly reduced the rate of false negatives (missed ODD violations) compared to a system without a monitor.
- State-based monitors demonstrated a noticeable decrease in false negatives compared to stateless monitors.
- In their case study on lane keeping, state-based monitors helped reduce the vehicle's cross-track error (CTE) range, bringing it closer to the desired safe range. For instance, one state-based monitor (Mon4) achieved a CTE range of [-0.89, 0.68], outperforming stateless monitors.
- Stateless monitors introduced an average delay of 201 µs per computation step, while state-based monitors increased this to 315 µs.
Limitations and Future Work
- Feature Engineering: The effectiveness of the learned monitors depends heavily on the choice of observable features used for learning. The paper suggests future work will explore this aspect more deeply.
- Learning Parameters: Finding the optimal length of input sequences, prediction horizon, and features is crucial and requires careful tuning.
- Generalizability: While evaluated on a lane-keeping task, extending the framework to other autonomous system components and domains is a natural next step.
- Counterexample Generation: The authors plan to explore more advanced counterexample-guided learning methods to obtain better counterexamples in each learning iteration.
Summary updated