Abstract
We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multi-agent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator.
Introduction
In safety-critical fields like autonomous driving, rule-based algorithms remain the industry standard because they are explainable and predictable. However, as these systems grow more complex, finding every possible "edge case" where the rules might fail becomes a needle-in-a-haystack problem.
Standard testing often relies on manual scenario creation or simple adversarial reinforcement learning (RL). The problem with basic adversarial RL is that it often produces "trivial" failures—for example, multiple cars intentionally ramming the player at once. These "kamikaze" attacks are unhelpful for debugging because no rule-based system can survive them. The challenge is to find natural but adversarial scenarios: situations where other drivers behave mostly normally but create a specific set of conditions that cause the rule-based agent to make a mistake.
Methodology
The authors propose FAILMAKER-ADVRL, built on top of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. To move beyond trivial attacks, they introduced four key innovations:
- Adversarial Learning with Personal Reward: NPCs aren't just "attackers." They are given their own goals (like reaching a destination quickly and safely). They only receive an adversarial bonus if they cause the player to fail without failing themselves. This forces them to be "clever" rather than suicidal.
- Contributor Identification (CI): In a multi-car simulation, it’s not always clear which car caused a crash. The system re-runs the simulation with different subsets of NPCs to identify the "true" culprits of a failure.
- Adversarial Reward Allocation (AdvRA): Once the culprits are identified, the adversarial reward is distributed specifically to the state-action pairs that most directly contributed to the player's failure, often based on proximity.
- Prioritized Sampling by Replay Buffer Partition (PS-RBP): If a rule-based agent is mostly good, failures are rare. To prevent these rare events from being "washed out" during training, the system maintains two separate replay buffers—one for successes ($D^+$) and one for failures ($D^-$)—and ensures the RL agents learn from both in a balanced way.
Experimental Setup
The researchers evaluated their method in two distinct environments:
- Simple Multi-agent Particle Environment: A 2D world where agents move toward goals while avoiding collisions. This served as a proof-of-concept for $N=1$ and $N=3$ adversarial NPCs.
- Microsoft AirSim: A high-fidelity 3D autonomous driving simulator. They used a "Coastline" environment with a two-lane road to test the system's ability to find failures in a more realistic, physics-based setting.
Results and Discussion
The authors compared FAILMAKER-ADVRL against several baselines, including "Attacker" (purely adversarial) and "P-Adv" (adversarial with personal rewards but without the specialized credit assignment).
Quantitative Performance
In the particle environment, FAILMAKER-ADVRL induced player failures in 95.6% to 99.2% of test cases, significantly higher than standard adversarial methods. In AirSim, the method achieved a 78.1% to 84.5% failure rate. Crucially, while the "Attacker" baseline also had high failure rates, it resulted in the NPCs themselves crashing nearly 100% of the time. FAILMAKER-ADVRL kept NPC failure rates low (around 3%), meaning the scenarios were realistic.
Key Insights
- The "Naturalness" Constraint: The most useful failure cases are those where the environment looks "normal." By penalizing NPCs for their own collisions, the authors ensure the discovered failures represent realistic traffic risks rather than impossible-to-avoid glitches.
- Solving the Credit Assignment Problem: In complex multi-agent scenes, "who caused the accident?" is a hard question. The use of simulation re-runs (CI) to isolate variables is a highly effective way to provide a clean training signal to the RL adversaries.
- Overcoming Sparse Rewards: Because failures are rare, standard RL struggles to learn. Partitioning the replay buffer acts as a form of "automated curriculum," forcing the model to study its rare "victories" (the player's failures) just as much as the common scenarios.
Conclusion
The authors acknowledge a few areas for growth. The current model assumes the agents have perfect knowledge of the state and does not yet account for failures in computer vision or sensor fusion. Future work needs to include more complex road layouts, such as intersections and traffic signs. Finally, the next logical step is an automated pipeline that uses these discovered failures to automatically retrain and harden the rule-based agent.
Summary updated