Practical Solutions for Machine Learning Safety in Autonomous Vehicles

Sina Mohseni, Mandar Pitale, Vasu Singh, Zhangyang Wang

Introduction

The rapid advancement of autonomous vehicles (AVs) relies heavily on Deep Neural Networks (DNNs) for perception and control. However, traditional automotive safety standards, primarily ISO 26262, were designed for deterministic systems where failures stem from hardware malfunctions or software bugs. Machine Learning (ML) models introduce a new class of "unsafe-unknown" failures—scenarios where the model is technically "bug-free" but fails due to environmental shifts, overconfidence, or adversarial inputs. This paper explores practical strategies to bridge this gap by integrating ML-specific safety measures into the established automotive engineering lifecycle.

The Safety Gap: Beyond ISO 26262

Traditional safety is managed via the "V-model," a rigorous process of design, implementation, and verification. While effective for standard software, it struggles with the probabilistic nature of AI. The industry has responded with ISO/PAS 21448, also known as SOTIF (Safety of the Intended Functionality). SOTIF acknowledges that a system can be unsafe even without a component failure, focusing instead on performance limitations and environmental triggers.

The augmented V-model integrating SOTIF (Safety of the Intended Functionality) alongside traditional ISO 26262 processes.

As shown in the augmented V-model, safety for AVs now requires a parallel track: SOTIF HARA (Hazard Analysis and Risk Assessment) and SOTIF Safety Concepts must run alongside traditional functional safety to address the inherent unpredictability of ML components.

Monitoring and Robustness Strategies

The authors propose a dual-pillar framework to enhance ML safety: Safe Fail monitoring and Safety Margin robustness.

1. Monitoring Functions (Safe Fail)

The goal of "Safe Fail" is to detect imminent failures in real-time, allowing the vehicle to transition to a "minimal risk condition," such as pulling over or alerting the driver.

  • Uncertainty Estimation: Techniques like Monte Carlo (MC) Dropout or Deep Ensembles allow a model to quantify its own confidence. If a model is "unsure" about a pedestrian's location, the system can trigger a safety intervention.
  • Out-of-Distribution (OOD) Detection: Models often fail when encountering data significantly different from their training set (e.g., a rare weather event). By implementing specialized "rejector nodes," models can identify these OOD samples and refuse to make a prediction.
A two-step training approach for Out-of-Distribution (OOD) detection, incorporating a 'reject' function to improve reliability.

2. Algorithm Robustness (Safety Margins)

Robustness ensures that models remain accurate despite environmental noise.

  • Domain Generalization: This involves training models to focus on "invariant features"—characteristics that remain constant across different lighting conditions or camera angles.
  • Data Augmentation: Using synthetic data and style transfer "hardens" the model against sensor noise, lens blur, or extreme weather conditions like heavy snow.

Key Insights and Results

The research highlights that accuracy is not a proxy for safety. A model with 99% accuracy can still be catastrophically overconfident in the remaining 1% of cases. Safety engineering requires prioritizing reliability over raw performance.

A significant contribution of this work is the mapping of high-level safety strategies to specific algorithmic opportunities. For instance, while interpretability tools like saliency maps help engineers understand model behavior, they are currently insufficient for the strict "traceability" required by safety audits. Furthermore, the authors demonstrate that adding a "reject" option significantly improves model reliability, though it reduces "test coverage" (the frequency with which the model provides an output).

Challenges and Future Outlook

Despite these advancements, several hurdles remain:

  • Computational Latency: Many effective safety monitors, such as Deep Ensembles, require multiple passes through a network, which can be too slow for high-speed real-time applications.
  • Adversarial Security: Defending against intentional, malicious perturbations remains an "arms race" with no definitive solution.
  • Formal Verification: Proving that a high-dimensional DNN will always behave correctly is currently an NP-hard problem, making absolute guarantees elusive.

In conclusion, the path to safe autonomous driving lies in a hybrid approach: combining the rigorous process of traditional automotive engineering with modern, real-time ML monitoring and robustness techniques.

Summary updated

BibTeX
@inproceedings{mohseni2020practical,
  author = {Sina Mohseni and Mandar Pitale and Vasu Singh and Zhangyang Wang},
  title = {{Practical Solutions for Machine Learning Safety in Autonomous Vehicles}},
  booktitle = {SafeAI @ AAAI},
  year = {2020},
}
Figure