LLM-Attacker: Enhancing Closed-Loop Adversarial Scenario Generation for Autonomous Driving With Large Language Models

Yuewen Mei, Tong Nie, Jian Sun, Ye Tian

TL;DR

This paper introduces LLM-Attacker, a novel framework that leverages Large Language Models (LLMs) to automatically generate challenging, safety-critical scenarios for training and testing autonomous driving systems (ADS). By using LLMs to identify and orchestrate "attacker" vehicles, LLM-Attacker creates more dangerous scenarios than existing methods, leading to ADS that are significantly safer and more robust.

What Problem Does This Paper Address?

Autonomous Driving Systems (ADS) are rapidly advancing, but ensuring their safety, especially in rare but critical accident scenarios, remains a major hurdle. These safety-critical events are infrequent in real-world data, making it difficult to train and test ADS effectively. Existing methods for generating these scenarios often struggle with identifying the right "attacker" vehicles within complex traffic, and they haven't fully explored how these generated scenarios can continuously improve the ADS itself. The paper aims to solve these limitations by creating a more intelligent and adaptive adversarial scenario generation system.

Methods

LLM-Attacker employs a multi-agent LLM-based approach for adversarial scenario generation. The core innovation lies in using LLMs for two key tasks:

  1. Adversarial Vehicle Identification: Instead of relying on manual design or simple rules, multiple LLM agents collaborate to identify which background vehicles are best positioned to act as attackers. This involves an iterative process where an "Initialization" module generates an initial identification function, an "Adversarial Simulation" tests its effectiveness, a "Reflection" module analyzes the results and suggests improvements, and a "Modification" module refines the function code. This agentic framework allows for adaptive and efficient identification of attackers in complex scenarios.
  2. Closed-Loop Trajectory Generation and Training: Once attackers are identified, their future trajectories are optimized to maximize the probability of a collision with the ego vehicle (the ADS being tested). This process is "closed-loop" because the generated adversarial scenarios are then used to retrain the ADS. As the ADS policy improves, the LLM-Attacker framework adapts by generating even more challenging scenarios, creating a continuous feedback loop for enhancing ADS robustness.

Datasets

The experiments utilize the Waymo Open Motion Dataset (WOMD), a collection of diverse real-world traffic scenarios. This dataset was used to extract 385 normal scenarios for training and 100 for testing. The paper does not introduce a new benchmark dataset but rather uses existing real-world data to generate novel adversarial scenarios.

Key Insights and Ideas

  • LLMs as Intelligent "Attackers": The paper demonstrates that LLMs can go beyond simple rule-based or random identification of attackers. Their ability to understand complex interactions and context allows them to pinpoint more effective adversarial agents, leading to more realistic and dangerous scenarios.
  • The Power of Collaboration: The multi-agent LLM framework, where different LLM agents specialize in initialization, reflection, and modification, is crucial for the iterative refinement of the attacker identification process. This collaborative approach enhances the intelligence and adaptability of the system.
  • Closed-Loop for Continuous Improvement: The most significant insight is the effectiveness of the closed-loop design. By continuously updating the ADS based on the generated adversarial scenarios, the system not only tests but actively improves the ADS's safety and robustness over time.
  • Multi-Attacker Advantage: The research highlights that scenarios with multiple attackers (specifically two) offer a good balance between increased attack effectiveness and manageable computational cost, leading to more robust ADS.

Key Results

  • Higher Attack Success Rate: LLM-Attacker achieved the highest adversarial attack success rate across various configurations (one to five attackers) compared to random, minimum Time-to-Collision (TTC), and Kinetic Field identification methods. For instance, with five attackers, LLM-Attacker achieved a 93.54% success rate.
  • Improved ADS Safety: ADS trained with LLM-generated adversarial scenarios (LLM-II) showed a significantly lower crash rate. Specifically, it achieved a crash rate of 8.81% with one attacker and 18.00% with two attackers, which is roughly half the crash rate of ADS trained only on normal scenarios (22.61%).
  • Enhanced Robustness: While the crash rate was reduced, the route completion rate remained competitive, indicating that the ADS trained with LLM-Attacker is not overly conservative but rather more capable of handling challenging situations.
  • Realistic Behavior: The generated adversarial behaviors exhibited low abnormal jerk rates and relatively low acceleration KL divergence, suggesting that the attackers' actions are realistic and not overly erratic.
  • Computational Efficiency: The LLM-based identification process, while iterative, was found to be computationally feasible, with an average cost of 4.42 hours for multiple rounds of identification and simulation.

Limitations and Future Work

  • LLM Performance Variability: The paper acknowledges that LLMs can produce outputs with significant differences in quality. While they employ strategies like "Best-of-N" sampling, further work could explore more robust methods to ensure consistent high-quality generation.
  • Generalizability of LLMs: The authors suggest comparing newly released LLMs to assess their capabilities in adversarial scenario generation, including reasoning efficiency and adversarial effectiveness.
  • Incorporating Traffic Rules: Future work could involve incorporating explicit traffic rule constraints into the attacker identification and trajectory generation processes to create even more diverse and edge-case scenarios.
  • Exploring Different Attack Strategies: The current focus is on identifying attackers and optimizing trajectories. Future research could explore more sophisticated attack strategies beyond simple collision maximization.

Summary updated

Abstract

Ensuring and improving the safety of autonomous driving systems (ADS) is crucial for deployment of highly automated vehicles, especially in safety-critical events. To address the rarity issue, adversarial scenario generation methods are developed, in which behaviors of traffic participants are manipulated to induce safety-critical events. However, existing methods still face two limitations. First, identification of the adversarial participant directly impacts the effectiveness of the generation. However, complexity of real-world scenarios, with numerous participants and diverse behaviors, makes identification challenging. Second, potential of generated safety-critical scenarios to continuously improve ADS performance remains underexplored. To address these issues, we propose LLM-attacker: a closed-loop adversarial scenario generation framework leveraging large language models (LLMs). Specifically, multiple LLM agents are designed and coordinated to identify optimal attackers. Then, the trajectories of attackers are optimized to generate adversarial scenarios. These scenarios are iteratively refined based on the performance of ADS, forming a feedback loop to improve ADS. Experimental results show that LLM-attacker can create more dangerous scenarios than other methods, and the ADS trained with it achieves a collision rate half that of training with normal scenarios. This indicates the ability of LLM-attacker to test and enhance the safety and robustness of ADS. The framework’s closed-loop design enables continuous scenario evolution compliant with regulatory standards, supporting both safety assurance and policy verification for ADS. Video demonstrations are provided at: https://drive.google.com/file/d/15rROV_ 8LUcc2jXSuSNBHVOHMCFKSn__B/view
BibTeX
@article{mei2025llm,
  author = {Yuewen Mei and Tong Nie and Jian Sun and Ye Tian},
  title = {{LLM-Attacker: Enhancing Closed-Loop Adversarial Scenario Generation for Autonomous Driving With Large Language Models}},
  journal = {IEEE Transactions on Intelligent Transportation Systems},
  year = {2025},
  doi = {10.1109/tits.2025.3578383},
}
Figure