TARGET: Traffic Rule-based Test Generation for Autonomous Driving Systems

Yao Deng, Jiaohong Yao, Zhi Tu, Xi Zheng, Mengshi Zhang, Tianyi ZhangarXiv 2023 · arXiv · OpenReview

Abstract

Recent accidents involving self-driving cars call for extensive testing efforts to improve the safety and robustness of autonomous driving. However, constructing test scenarios for autonomous driving is tedious and time-consuming. In this work, we develop an end-to-end test generation framework called TARGET, which automatically constructs test scenarios from human-written traffic rules in an autonomous driving simulator. To handle the ambiguity and sophistication of natural language, TARGET uses GPT-3 to extract key information related to the test scenario from a traffic rule and represents the extracted information in a test scenario schema. Then, TARGET synthesizes the corresponding scenario scripts to construct the test scenario based on the scenario representation. We have evaluated TARGET on four autonomous driving systems, 18 traffic rules, and 8 road maps. TARGET can successfully generate 75 test scenarios and detect 247 traffic rule violations. Based on the violation logs (e.g., waypoints of ego vehicles), we are able to identify three underlying issues in these autonomous driving systems, which are either confirmed by the developers or the existing bug reports.

Introduction

Testing Autonomous Driving Systems (ADSs) remains a significant bottleneck in the path toward full vehicle autonomy. While high-fidelity simulators like CARLA offer safe environments for evaluation, the manual creation of test scenarios is labor-intensive. Industry-standard languages such as OpenSCENARIO are notoriously verbose; a simple maneuver can require hundreds of lines of code. Furthermore, Large Language Models (LLMs), while capable of understanding natural language traffic rules, often struggle to generate executable simulation scripts directly, frequently "hallucinating" map coordinates or failing to adhere to strict syntax requirements.

The TARGET Framework

To bridge the gap between high-level traffic laws (e.g., "Yield to vehicles in a roundabout") and executable simulation code, we introduce TARGET (TrAffic Rule-based test GEneraTion). TARGET employs a "divide and conquer" strategy that decouples the logical reasoning of traffic rules from the concrete geometric requirements of the simulator.

The TARGET framework architecture, illustrating the flow from natural language traffic rules to executable test reports.

The framework consists of four primary components: a Scenario DSL that defines the environment and actors, a Rule Parser that converts natural language into this DSL, a Scenario Generator that maps logical routes to simulator coordinates, and a Scenario Monitor that logs safety violations during execution.

Multi-Stage LLM Parsing

Directly generating simulator scripts with LLMs often leads to syntax errors and logical inconsistencies. TARGET addresses this by using a three-stage prompting pipeline to extract logic into a custom, high-level Domain-Specific Language (DSL). This DSL focuses on four modular components: Environment (weather/time), Road Network (intersection types), Actors (behaviors), and Oracle (safety rules).

The multi-stage LLM pipeline designed to ensure knowledge extraction, validation, and syntax alignment.

The pipeline ensures accuracy through:

  1. Knowledge Extraction: Identifying core facts from the traffic rule.
  2. Knowledge Validation: A self-check phase where the LLM verifies its output against the source text.
  3. Syntax Alignment: Normalizing the output to match a predefined vocabulary (e.g., converting "drive straight" to the standard "go forward").

Map-Based Route Search and Execution

Because LLMs lack inherent knowledge of a simulator's specific map geometry, TARGET utilizes a hierarchical map-based route search algorithm. This algorithm searches the simulator's map metadata to find specific coordinates and paths that satisfy the logical requirements of the DSL (e.g., finding a T-junction where an NPC can approach from the right). Once the scenario is generated, the Scenario Monitor tracks the ADS's performance, logging rule violations, collisions, and timeouts.

Experimental Results

TARGET was evaluated using 54 rules from the Texas Driver Handbook across multiple simulators (CARLA, LGSVL, MetaDrive) and industrial-grade ADS stacks like Baidu Apollo 7.0 and Autoware.

Key findings include:

  • High Parsing Accuracy: The GPT-4 powered parser achieved 91.8% accuracy in DSL conversion.
  • Bug Discovery: Across 284 scenarios, TARGET uncovered 610 erroneous behaviors. For instance, Apollo 7.0 showed difficulties detecting slow-moving pedestrians, while Autoware suffered from localization errors leading to roadside collisions.
  • Developer Confirmation: Several identified issues were confirmed by ADS developers, including a known limitation in perception functionality.

Conclusion

TARGET demonstrates that abstraction is a powerful tool for LLM-based testing. By using a high-level DSL as an intermediary, the framework leverages the LLM's reasoning strengths while delegating precise mathematical calculations to traditional algorithms. Future work will focus on handling complex sequential logic and incorporating multi-modal LLMs to interpret traffic rule diagrams alongside text.

Summary updated

Abstract

Recent accidents involving self-driving cars call for extensive testing efforts to improve the safety and robustness of autonomous driving. However, constructing test scenarios for autonomous driving is tedious and time-consuming. In this work, we develop an end-to-end test generation framework called TARGET, which automatically constructs test scenarios from human-written traffic rules in an autonomous driving simulator. To handle the ambiguity and sophistication of natural language, TARGET uses GPT-3 to extract key information related to the test scenario from a traffic rule and represents the extracted information in a test scenario schema. Then, TARGET synthesizes the corresponding scenario scripts to construct the test scenario based on the scenario representation. We have evaluated TARGET on four autonomous driving systems, 18 traffic rules, and 8 road maps. TARGET can successfully generate 75 test scenarios and detect 247 traffic rule violations. Based on the violation logs (e.g., waypoints of ego vehicles), we are able to identify three underlying issues in these autonomous driving systems, which are either confirmed by the developers or the existing bug reports.
BibTeX
@misc{deng2023target,
  author = {Yao Deng and Jiaohong Yao and Zhi Tu and Xi Zheng and Mengshi Zhang and Tianyi Zhang},
  title = {{TARGET: Traffic Rule-based Test Generation for Autonomous Driving Systems}},
  year = {2023},
  eprint = {2305.06018},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.06018},
}
Figure