Runtime Monitors for Operational Design Domains of Black-Box ML-Models

Hazem Torfah, Sanjit A. SeshiaMLSW @ NeurIPS 2022 · OpenReview

TL;DR

This paper introduces a framework for learning "runtime monitors" that ensure autonomous systems, particularly those using black-box machine learning (ML) components, stay within their safe operational design domain (ODD). By learning from simulations and using counterexample-guided refinement, these monitors predict when a system is about to exit its safe operating conditions, enabling a switch to a fallback safety controller.

What Problem Does This Paper Address?

Autonomous systems, like self-driving cars, increasingly rely on complex ML models for critical tasks like perception and control. However, these ML models can be brittle, failing unexpectedly when encountering situations outside their training data distribution (the "distribution gap"). This brittleness poses a significant safety risk. While the concept of an Operational Design Domain (ODD) aims to define safe operating conditions, simply defining an ODD isn't enough; it needs to be actively monitored at runtime. The challenge is to create monitors for ODDs of "black-box" ML components, where we don't have direct access to their internal workings or training data, and where the ODD itself might not be directly observable.

Methods

The authors propose a simulation-based framework that integrates with VERIFAI, an open-source toolkit for formal system analysis. The core idea is to:

  1. Simulation-based Analysis: Use VERIFAI and SCENIC (a probabilistic programming language) to generate diverse simulation scenarios of the autonomous system in its environment.
  2. Data Generation and Learning: Collect "traces" of observable features from these simulations. These traces are then processed (filtered, transformed) to create a training dataset. A learner then trains a "monitor" based on this data. The key is that the ODD is learned as a "monitorable" property, meaning it can be checked using observable features.
  3. Counterexample-Guided Refinement: The framework uses a conformance tester that actively searches for "counterexamples" – situations where the learned monitor incorrectly predicts safety or danger. These counterexamples are fed back to the data generation and learning components to refine the monitor iteratively.
  4. Monitor Types: They investigate two types of monitors:
  5. Stateless monitors: Make predictions based on the current observation of features.
  6. State-based monitors: Make predictions based on a sequence of recent observations, allowing them to capture temporal dynamics.

Datasets

The paper uses simulations generated by the CARLA simulator [5] and modeled using SCENIC [8]. The training and testing data are derived from these simulations, not from pre-existing, publicly released datasets for this specific ODD monitoring task. The simulations cover various weather conditions, road types, initial positions, and sun angles.

Key Insights and Ideas

  • Monitorability is Crucial: Simply defining an ODD is insufficient for safety-critical systems. The ODD must be expressible in terms of observable features and implementable as a runtime monitor.
  • State-based Monitors Outperform Stateless Ones: Capturing the temporal evolution of system states (using sequences of observations) significantly improves the ability of monitors to detect potential ODD violations, leading to fewer false negatives.
  • Counterexample-Guided Learning is Effective: Iteratively refining the monitor based on counterexamples found during conformance testing is key to learning accurate and robust ODD monitors.
  • Balancing Complexity and Delay: While state-based monitors offer better safety, they introduce computational delay. The paper highlights the need to balance the length of input sequences for the monitor with the acceptable latency.

Key Results

  • The learned monitors (both stateless and state-based) significantly reduced the rate of false negatives (missed ODD violations) compared to a system without a monitor.
  • State-based monitors demonstrated a noticeable decrease in false negatives compared to stateless monitors.
  • In their case study on lane keeping, state-based monitors helped reduce the vehicle's cross-track error (CTE) range, bringing it closer to the desired safe range. For instance, one state-based monitor (Mon4) achieved a CTE range of [-0.89, 0.68], outperforming stateless monitors.
  • Stateless monitors introduced an average delay of 201 µs per computation step, while state-based monitors increased this to 315 µs.

Limitations and Future Work

  • Feature Engineering: The effectiveness of the learned monitors depends heavily on the choice of observable features used for learning. The paper suggests future work will explore this aspect more deeply.
  • Learning Parameters: Finding the optimal length of input sequences, prediction horizon, and features is crucial and requires careful tuning.
  • Generalizability: While evaluated on a lane-keeping task, extending the framework to other autonomous system components and domains is a natural next step.
  • Counterexample Generation: The authors plan to explore more advanced counterexample-guided learning methods to obtain better counterexamples in each learning iteration.

Summary updated

Abstract

Autonomous systems are increasingly relying on machine learning (ML) components to perform a variety of complex tasks in perception, prediction, and control.To guarantee the safety of ML-based autonomous systems, it is important to capture their operational design domain (ODD), i.e., the conditions under which using the ML components does not endanger the safety of the system. In this paper, we present a framework for learning runtime monitors for ODDs of autonomous systems with black-box ML components. A runtime monitor of an ODD predicts based on a sequence of monitorable observations whether the system is about to exit its ODD. We particularly investigate the learning of optimal monitors based on counterexample-guided refinement and conformance testing. We evaluate our approach on a case study from the domain of autonomous driving.
BibTeX
@inproceedings{torfah2022runtime,
  author = {Hazem Torfah and Sanjit A. Seshia},
  title = {{Runtime Monitors for Operational Design Domains of Black-Box ML-Models}},
  booktitle = {MLSW @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=6_AtjSBhqx},
}
Figure