Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving

Akifumi Wachi

Abstract

We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multi-agent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator.

Introduction

In safety-critical fields like autonomous driving, rule-based algorithms remain the gold standard due to their explainability and predictability. However, as these systems grow in complexity, identifying all potential failure modes becomes a "needle in a haystack" problem. Traditional adversarial testing often yields trivial failures—such as "kamikaze" maneuvers where an NPC simply rams the player—or suffers from sparse data, where the rarity of failures in a well-designed system makes it difficult for an AI to learn effective adversarial policies.

The FAILMAKER-ADVRL Framework

To address these challenges, we introduce FAILMAKER-ADVRL, a framework built upon Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that incorporates three novel components to generate realistic and informative failure scenarios.

The FAILMAKER-ADVRL framework utilizes partitioned replay buffers and contributor identification to efficiently train adversarial NPCs.

Key Components

  1. Adversarial Learning with Personal Reward: NPCs are given a dual objective: a personal reward for reaching their own destination safely and an adversarial reward triggered only by a player failure. This constraint ensures that NPCs create dangerous situations while maintaining "natural" traffic behavior.
  2. Contributor Identification (CI) and Adversarial Reward Allocation (AdvRA): To solve the credit assignment problem in multi-agent crashes, the system re-runs simulations with subsets of NPCs. A distance-based contribution function then distributes the adversarial reward only to the agents that actively participated in setting the trap.
  3. Prioritized Sampling by Replay Buffer Partition (PS-RBP): To overcome the sparsity of failure events, the framework maintains separate replay buffers for successes ($D^+$) and failures ($D^-$). During training, the system intentionally oversamples from $D^-$ to accelerate learning from rare adversarial successes.

Experimental Results

The framework was evaluated in the Microsoft AirSim simulator using a high-fidelity coastline environment. We compared FAILMAKER-ADVRL against several baselines, including a pure "Attacker" (ignoring its own safety) and standard adversarial RL.

Comparison of player failure probability across different adversarial training methods.

In the AirSim environment with two NPCs, FAILMAKER-ADVRL induced a player failure rate of 84.5%, significantly outperforming standard adversarial methods which achieved only 5-7%. While the pure "Attacker" baseline achieved a higher failure rate, it resulted in NPC crashes 98% of the time. In contrast, FAILMAKER-ADVRL maintained low NPC failure rates (approx. 3%), successfully generating "natural" dangerous scenarios where the adversaries survived the encounter.

Conclusion and Future Work

FAILMAKER-ADVRL demonstrates that constraining adversarial agents with personal safety goals leads to more useful and realistic failure scenarios. Future work will focus on bridging the perception gap—accounting for sensor errors rather than assuming perfect state knowledge—and increasing environmental complexity by including pedestrians, traffic lights, and intersections. Additionally, incorporating formal safety metrics like Time-to-Collision (TTC) could further refine reward allocation.

Summary updated

Abstract

We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multi-agent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator.
BibTeX
@inproceedings{wachi2019failure,
  author = {Akifumi Wachi},
  title = {{Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=FYPBgixjbeY},
}
Figure