Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

Jonathan Uesato, Ananya Kumar, Csaba Szepesvári, Tom Erez, Avraham Ruderman, Keith Anderson, Krishnamurthy Dj Dvijotham, Nicolas Heess, Pushmeet Kohli

Abstract

Standard random testing, such as Vanilla Monte Carlo (VMC), is mathematically incapable of reliably finding rare, catastrophic failures in safety-critical AI agents within a reasonable compute budget. This paper introduces an adversarial evaluation framework that utilizes a "continuation approach"—learning failure patterns from weaker, earlier versions of an agent—to uncover catastrophic bugs and estimate risk up to thousands of times faster than standard methods.

The Challenge of Rare Failures

How do you prove a self-driving car is safe if it only crashes once every 100 million miles?

In reinforcement learning (RL), agents are typically evaluated by running them in a simulator and averaging the results. However, for safety-critical systems, the "average" performance is far less important than the "worst-case" failure. If a failure occurs with probability $\epsilon$, you need at least $1/\epsilon$ test runs to have a reasonable chance of observing it even once.

For high-stakes applications, this makes rigorous evaluation computationally more expensive than the training process itself. Furthermore, because failures are so rare in a highly optimized, "finished" agent, there is almost no signal for standard optimization algorithms to find those specific breaking points. They become "black swans"—highly improbable but catastrophic events.

Methodology: An Adversarial Approach

The authors propose a two-pronged approach to solve this "needle in a haystack" problem:

  1. Adversarial Value Functions (AVF): The researchers train a predictor, $f(x)$, that estimates the probability of failure given a set of initial conditions $x$. This AVF acts as an "adversary" designed to search for the most difficult starting states.
  2. The Continuation Approach: Because the final agent rarely fails, training an AVF on it directly is nearly impossible due to a lack of failure data. Instead, the authors leverage historical training data. They train the AVF on earlier, "clumsier" versions of the agent that failed frequently. The core insight is that the failure modes of a weak agent often highlight underlying structural weaknesses that persist—albeit more subtly—in the final, robust agent.
  3. Risk Estimation via Importance Sampling: To ensure the evaluation isn't simply biased toward failures, the AVF is used to perform Importance Sampling. This focuses the evaluation on high-risk scenarios while mathematically re-weighting the results to provide an unbiased estimate of the true failure probability.

Experimental Setup

The researchers evaluated their approach on two simulated environments:

  • Driving (TORCS): A 3D racing simulator where the agent must navigate a track without hitting walls. A failure is defined as any collision.
  • Humanoid (MuJoCo): A complex 21-degree-of-freedom robot tasked with standing still. A failure is defined as the robot's head falling below a specific height threshold.

Key Insights and Results

The adversarial approach outperformed Vanilla Monte Carlo (VMC) by massive margins, proving that adversarial testing can be a rigorous statistical tool rather than just a way to "break" a system.

Error in risk estimate for Driving and Humanoid domains

As shown in the risk estimate error plots above, the AVF estimator converges to an accurate assessment of failure probability significantly faster than the VMC estimator in both the Driving and Humanoid domains.

  • Failure Search Speedup: The AVF adversary found catastrophic failures 198x faster in the Driving domain and 3,100x faster in the Humanoid domain compared to random testing.
  • Risk Estimation Efficiency: To achieve a reliable estimate of the failure rate, the AVF method required 14x to 34x fewer samples than VMC.
  • Wall-clock Time: On the Humanoid task, VMC would take roughly 3 days of constant simulation to find a single failure; the AVF method found one in just 6 minutes.
Table showing evaluation cost and acceleration factors

The table above highlights the stark contrast in evaluation costs. The acceleration factor demonstrates that the AVF approach is multiple orders of magnitude more efficient, fundamentally changing the feasibility of rigorous safety testing.

Additional Findings

  • Evaluation is harder than training: The authors demonstrated that even matching the total compute used for training is often insufficient to find a single failure using random testing.
  • Weaker agents are "canaries": You don't need to see the final agent fail to know where it might be weak. By observing where the agent struggled during early training, you can predict its vulnerabilities as a fully trained model.
  • Robustness isn't monotonic: The "final" agent isn't always the most robust one. The AVF method revealed that agents often become less robust to specific edge cases late in training—a phenomenon standard evaluation entirely misses.

Limitations and Future Work

While highly effective, the approach has limitations. The continuation approach relies on the "Relatedness" Assumption—it assumes the final agent fails in ways related to its earlier versions. If the final agent develops an entirely new failure mode that never appeared during training, this method might miss it.

Additionally, the current work focuses on finding adversarial starting states. Future research could explore "online" adversaries that dynamically perturb the environment or the agent's sensors throughout an episode. Finally, while the AVF is efficient, building an accurate failure predictor in extremely high-dimensional real-world data (like raw video feeds) remains a complex challenge.

Summary updated

Abstract

This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.
BibTeX
@inproceedings{uesato2019rigorous,
  author = {Jonathan Uesato and Ananya Kumar and Csaba Szepesvári and Tom Erez and Avraham Ruderman and Keith Anderson and Krishnamurthy Dj Dvijotham and Nicolas Heess and Pushmeet Kohli},
  title = {{Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=zbe4cWIK1-S},
}
Figure