Safety Assurance for Systems with Machine Learning Components

Chelsea SidraneAAAI 2021 · DOI

Abstract

As machine learning (ML) components are increasingly integrated into safety-critical systems such as autonomous aircraft and vehicles, the lack of formal reliability guarantees creates a significant "trust gap." This paper explores a multi-faceted framework for safety assurance, combining human-centric imitation learning, formal mathematical verification via the OVERT method, and adaptive stress testing. By addressing training safety, verification complexity, and testing uncertainty, this research provides a pathway toward certifying neural network controllers in complex, nonlinear environments.

Introduction

The transition of deep learning from low-stakes applications, such as image recognition, to safety-critical domains, such as autonomous flight, necessitates a shift in how we evaluate system reliability. Traditional ML models often operate as "black boxes," lacking the formal guarantees required for safety certification. This research identifies three primary gaps in the current lifecycle of safety-critical ML: training safety, verification complexity, and testing uncertainty. To bridge these gaps, we propose a three-pronged approach involving improved imitation learning, formal verification tools, and rigorous adversarial testing.

Human-Gate DAgger: Enhancing Training Safety

In imitation learning, frameworks like Dataset Aggregation (DAgger) are used to teach agents by having a human expert provide corrections. However, standard DAgger often suffers from abrupt control handovers between the human and the "novice" AI, leading to dangerous oscillations or placing the AI in control before it is sufficiently trained.

The proposed Human-Gate DAgger (HG-DAgger) addresses this by utilizing Bayesian deep learning to estimate the AI's confidence in its actions. Crucially, HG-DAgger shifts control authority back to the human expert. Instead of the algorithm deciding when to intervene, the human maintains the power to take over. These intervention points serve as a calibration mechanism: the moments a human chooses to intervene are used to define the safety thresholds for the AI’s confidence metric, effectively translating human intuition into a mathematical constraint.

OVERT: Formal Verification of Nonlinear Systems

Proving that a neural network controller will keep a vehicle within safe bounds is mathematically challenging, particularly when the vehicle dynamics are nonlinear and operate in discrete time steps.

The OVERT tool provides a solution through formal verification of "closed-loop" systems (the AI controller plus the vehicle). OVERT functions by creating a simplified "over-approximation" of the nonlinear system. This method relies on the property of soundness: if the tool can prove that the simplified, more inclusive version of the system is safe, then the original, more complex system is mathematically guaranteed to be safe as well. This approach bypasses the need to solve exact nonlinear equations, which is often computationally impossible, while still providing rigorous safety guarantees.

Adaptive Stress Testing and Coverage

Even with verification, systems must be tested against edge cases. Adaptive Stress Testing (AST) employs an adversarial reinforcement learning agent to intentionally search for system failures.

A persistent problem in testing is determining when a system has been tested "enough." This research treats failure detection as a search problem. By applying Bayesian RL to the adversarial agent, we can measure "testing coverage." This allows engineers to move beyond the mere absence of found bugs toward a statistical confidence—for example, being 99% certain that no failures exist within a specific operational region.

Key Insights

The integration of these methods yields several critical insights for safety assurance:

  1. Human-in-the-loop Calibration: Human judgment is not just a backup but a ground-truth source for defining safety boundaries in high-dimensional state spaces.
  2. Soundness through Approximation: Rigorous safety does not require perfect modeling of every nonlinear detail; it requires modeling that "envelopes" all possible real-world behaviors.
  3. Quantifiable Testing: Testing can be transformed from an ad-hoc process into a rigorous search task with quantifiable coverage metrics.

Results and Future Work

Experimental results demonstrate that HG-DAgger successfully mitigates unsafe oscillations by respecting human control authority. Furthermore, the OVERT method has proven more computationally efficient than general-purpose nonlinear verification tools, producing tighter approximations for discrete-time systems.

Future research aims to extend these methods in several directions. To address the limitation of fixed-time windows in verification, work is underway to incorporate Lyapunov stability into OVERT, which would guarantee safety over an infinite time horizon. Additionally, to manage the computational burden of verifying large-scale neural networks, researchers are exploring techniques to approximate large networks with smaller, more verifiable versions. Finally, the reliability of Bayesian confidence estimates remains a central focus to ensure the integrity of testing coverage metrics.

Summary updated

BibTeX
@inproceedings{sidrane2021safety,
  author = {Chelsea Sidrane},
  title = {{Safety Assurance for Systems with Machine Learning Components}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17864},
}
Figure