Abstract
This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.
Introduction
As reinforcement learning (RL) agents are increasingly considered for safety-critical applications like autonomous driving, standard evaluation metrics based on average-case performance become insufficient. A catastrophic failure occurring once every 100 million miles is unacceptable, yet such rare events are difficult to detect using standard methods. The core challenge lies in rare event estimation. Vanilla Monte Carlo (VMC) evaluation relies on random sampling, which is computationally expensive and can provide a false sense of security if failures are not encountered during testing.
Methodology: Adversarial Evaluation
To address the limitations of random sampling, we propose an adversarial evaluation framework centered on a Failure Probability Predictor (AVF).
The Failure Probability Predictor (AVF)
The AVF is a model $f(x)$ designed to estimate the probability of agent failure given initial conditions $x$. However, training such a model from scratch is difficult because failures are rare in high-performing agents, leading to a lack of training signal.
The Continuation Approach
We introduce a "continuation" strategy to overcome the data scarcity problem. Instead of generating new data, we leverage the agent's own training history. Early in training, agents are less robust and fail frequently. This historical data provides a rich signal for learning the characteristics of "difficult" scenarios that might still trigger failures in the final, more robust agent.
Risk Estimation via Importance Sampling
To ensure unbiased risk estimates, we use Importance Sampling. The AVF guides the search toward high-risk initial conditions. The resulting failure rates are then mathematically re-weighted to provide an accurate estimate of the true failure probability, allowing for rigorous safety audits.
Experimental Evaluation
We evaluated our approach on two high-dimensional benchmarks:
- Driving (TORCS): A 3D simulator where failure is defined as a collision.
- Humanoid (MuJoCo): A complex robotic control task where failure is defined by the humanoid falling.
Results
Our adversarial approach demonstrates significant improvements in both failure search and risk estimation efficiency.
Efficiency Gains
As shown in the experimental results, the AVF-based method identifies catastrophic failures orders of magnitude faster than random testing. In the Humanoid domain, the acceleration factor reached up to 3800x, reducing evaluation time from days to minutes.
Key Insights
- Training History Utility: Mistakes made during the learning process are highly predictive of the rare failures of the final agent.
- Robustness Fluctuations: Agent robustness does not always improve monotonically with average reward. Adversarial evaluation can uncover new blind spots created during the optimization process.
Conclusion and Future Work
Adversarial evaluation provides a rigorous path toward certifying the safety of RL agents. While the continuation approach is highly effective, it may miss "novel" failure modes that never appeared during training. Future research will explore extending this framework to continuous safety metrics and real-world hardware-in-the-loop testing.
Summary updated