Abstract
Deployment of modern data-driven machine learning methods, most often realized by deep neural networks (DNNs), in safety-critical applications such as health care, industrial plant control, or autonomous driving is highly challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability and implausible predictions to directed attacks by means of malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from so-called safety concerns, properties that preclude their deployment as no argument or experimental setup can help to assess the remaining risk. In recent years, an abundance of state-of-the-art techniques aiming to address these safety concerns has emerged. This chapter provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our work addresses machine learning experts and safety engineers alike: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern machine learning methods. We hope that this contribution fuels discussions on desiderata for machine learning systems and strategies on how to help to advance existing approaches accordingly.
Introduction: The Safety Gap in Deep Learning
While Deep Neural Networks (DNNs) have achieved superhuman performance in specialized tasks, their integration into safety-critical applications—such as autonomous driving, healthcare, and industrial control—remains a significant challenge. These models are often "black boxes," characterized by inherent vulnerabilities that traditional software engineering is ill-equipped to handle. They may fail under distribution shifts, provide overconfident yet incorrect predictions, or succumb to adversarial attacks.
In high-stakes environments, high average accuracy is insufficient. Engineers require rigorous methods to quantify risks, interpret decisions, and implement real-time safety mechanisms. This survey bridges the gap between academic research and safety engineering by providing a structured framework: Inspect, Understand, and Overcome.
A Taxonomy for AI Safety
The authors categorize the vast landscape of AI safety research into a practical lifecycle designed to identify and mitigate model-immanent insufficiencies.
1. Dataset Optimization and Inspection
Safety begins with the data. This category includes outlier and anomaly detection to identify Out-of-Distribution (OOD) data that the model was not trained to handle. Techniques such as active learning and automated corner-case detection help engineers find the "blind spots" in their training sets, ensuring the model is exposed to the rare but critical scenarios it will encounter in the real world.
2. Robust Training and Mitigation
To make models resilient to both natural hazards (corruption robustness) and malicious inputs (adversarial robustness), several training strategies are employed:
- Adversarial Training: Incorporating perturbed examples into the training loop to harden the model.
- Hyperparameter Optimization: Tuning the architecture and training process to prioritize stability over raw accuracy.
- Loss Function Modification: Penalizing model instability or overconfidence during the optimization phase.
3. Interpretability and Understanding
Interpretability tools allow engineers to "peek inside" the DNN. Pixel attribution methods (saliency maps) highlight which parts of an input influenced a decision, helping to detect "Clever Hans" predictors—models that rely on irrelevant artifacts (like a watermark or a specific background) rather than the actual object of interest. Other methods include visual analytics and the use of interpretable proxies, where a complex DNN is approximated by a simpler, rule-based model for easier verification.
4. Uncertainty Estimation
A safe model must know when it doesn't know. Moving beyond point estimates, safety-critical systems use probabilistic outputs.
- Bayesian Neural Networks and MC Dropout: These provide a measure of model uncertainty, though Monte Carlo (MC) Dropout is often preferred for its scalability.
- Deep Ensembles: While computationally expensive, ensembles of multiple models typically provide the most diverse and reliable uncertainty estimates.
- Confidence Calibration: Ensuring that a model’s predicted probability (e.g., 90% confidence) actually matches its historical accuracy.
5. Verification, Validation, and Monitoring
Formal methods are used to prove that a model behaves correctly within specific bounds. Additionally, "black-box" observers can monitor model performance during deployment, acting as a safety layer that can trigger a fallback mode if the model's behavior becomes erratic.
Benchmarking Safety
The survey reviews these methods across standard benchmarks:
- Classification: CIFAR-10/100, ImageNet, and SVHN.
- Autonomous Driving: Cityscapes and KITTI for perception tasks.
- Simulation: The CARLA environment is highlighted as essential for generating synthetic corner cases that are too dangerous or rare to capture in the real world.
Key Insights and Results
The research highlights several critical takeaways for safety engineers:
- Safety vs. Security: There is a vital distinction between adversarial robustness (protection against attacks) and corruption robustness (protection against natural noise like rain or sensor failure). For road-legal systems, corruption robustness is often the primary bottleneck.
- The Overconfidence Trap: Modern DNNs are notoriously poor at calibration, often assigning near-certainty to incorrect labels. Calibration is a fundamental safety requirement, not just a performance metric.
- Efficiency vs. Robustness: While model compression (pruning and quantization) can reduce model size by 35x to 49x, its impact on robustness remains an active area of debate and research.
Conclusion: The Path Forward
Despite significant progress, several hurdles remain. Many safety methods do not yet scale to high-resolution, real-time tasks like semantic segmentation in moving vehicles. Furthermore, there is a lack of unified metrics for "good" explanations or "sufficient" testing.
The authors conclude that the next frontier is holistic safety argumentation. No single method is a "silver bullet." Instead, safety must be achieved through overlapping mechanisms—combining robust training with real-time anomaly detection and formal verification—to create a "tight safety argument" that satisfies both regulatory bodies and the public.
Summary updated