Abstract
We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multi-agent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator.
Introduction
In safety-critical fields like autonomous driving, rule-based algorithms remain the gold standard due to their explainability and predictability. However, as these systems grow in complexity, identifying all potential failure modes becomes a "needle in a haystack" problem. Traditional adversarial testing often yields trivial failures—such as "kamikaze" maneuvers where an NPC simply rams the player—or suffers from sparse data, where the rarity of failures in a well-designed system makes it difficult for an AI to learn effective adversarial policies.
The FAILMAKER-ADVRL Framework
To address these challenges, we introduce FAILMAKER-ADVRL, a framework built upon Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that incorporates three novel components to generate realistic and informative failure scenarios.
Key Components
- Adversarial Learning with Personal Reward: NPCs are given a dual objective: a personal reward for reaching their own destination safely and an adversarial reward triggered only by a player failure. This constraint ensures that NPCs create dangerous situations while maintaining "natural" traffic behavior.
- Contributor Identification (CI) and Adversarial Reward Allocation (AdvRA): To solve the credit assignment problem in multi-agent crashes, the system re-runs simulations with subsets of NPCs. A distance-based contribution function then distributes the adversarial reward only to the agents that actively participated in setting the trap.
- Prioritized Sampling by Replay Buffer Partition (PS-RBP): To overcome the sparsity of failure events, the framework maintains separate replay buffers for successes ($D^+$) and failures ($D^-$). During training, the system intentionally oversamples from $D^-$ to accelerate learning from rare adversarial successes.
Experimental Results
The framework was evaluated in the Microsoft AirSim simulator using a high-fidelity coastline environment. We compared FAILMAKER-ADVRL against several baselines, including a pure "Attacker" (ignoring its own safety) and standard adversarial RL.
In the AirSim environment with two NPCs, FAILMAKER-ADVRL induced a player failure rate of 84.5%, significantly outperforming standard adversarial methods which achieved only 5-7%. While the pure "Attacker" baseline achieved a higher failure rate, it resulted in NPC crashes 98% of the time. In contrast, FAILMAKER-ADVRL maintained low NPC failure rates (approx. 3%), successfully generating "natural" dangerous scenarios where the adversaries survived the encounter.
Conclusion and Future Work
FAILMAKER-ADVRL demonstrates that constraining adversarial agents with personal safety goals leads to more useful and realistic failure scenarios. Future work will focus on bridging the perception gap—accounting for sensor errors rather than assuming perfect state knowledge—and increasing environmental complexity by including pedestrians, traffic lights, and intersections. Additionally, incorporating formal safety metrics like Time-to-Collision (TTC) could further refine reward allocation.
Summary updated