Abstract
Recent accidents involving self-driving cars call for extensive testing efforts to improve the safety and robustness of autonomous driving. However, constructing test scenarios for autonomous driving is tedious and time-consuming. In this work, we develop an end-to-end test generation framework called TARGET, which automatically constructs test scenarios from human-written traffic rules in an autonomous driving simulator. To handle the ambiguity and sophistication of natural language, TARGET uses GPT-3 to extract key information related to the test scenario from a traffic rule and represents the extracted information in a test scenario schema. Then, TARGET synthesizes the corresponding scenario scripts to construct the test scenario based on the scenario representation. We have evaluated TARGET on four autonomous driving systems, 18 traffic rules, and 8 road maps. TARGET can successfully generate 75 test scenarios and detect 247 traffic rule violations. Based on the violation logs (e.g., waypoints of ego vehicles), we are able to identify three underlying issues in these autonomous driving systems, which are either confirmed by the developers or the existing bug reports.
Introduction
Testing Autonomous Driving Systems (ADSs) remains a significant bottleneck in the path toward full vehicle autonomy. While high-fidelity simulators like CARLA offer safe environments for evaluation, the manual creation of test scenarios is labor-intensive. Industry-standard languages such as OpenSCENARIO are notoriously verbose; a simple maneuver can require hundreds of lines of code. Furthermore, Large Language Models (LLMs), while capable of understanding natural language traffic rules, often struggle to generate executable simulation scripts directly, frequently "hallucinating" map coordinates or failing to adhere to strict syntax requirements.
The TARGET Framework
To bridge the gap between high-level traffic laws (e.g., "Yield to vehicles in a roundabout") and executable simulation code, we introduce TARGET (TrAffic Rule-based test GEneraTion). TARGET employs a "divide and conquer" strategy that decouples the logical reasoning of traffic rules from the concrete geometric requirements of the simulator.
The framework consists of four primary components: a Scenario DSL that defines the environment and actors, a Rule Parser that converts natural language into this DSL, a Scenario Generator that maps logical routes to simulator coordinates, and a Scenario Monitor that logs safety violations during execution.
Multi-Stage LLM Parsing
Directly generating simulator scripts with LLMs often leads to syntax errors and logical inconsistencies. TARGET addresses this by using a three-stage prompting pipeline to extract logic into a custom, high-level Domain-Specific Language (DSL). This DSL focuses on four modular components: Environment (weather/time), Road Network (intersection types), Actors (behaviors), and Oracle (safety rules).
The pipeline ensures accuracy through:
- Knowledge Extraction: Identifying core facts from the traffic rule.
- Knowledge Validation: A self-check phase where the LLM verifies its output against the source text.
- Syntax Alignment: Normalizing the output to match a predefined vocabulary (e.g., converting "drive straight" to the standard "go forward").
Map-Based Route Search and Execution
Because LLMs lack inherent knowledge of a simulator's specific map geometry, TARGET utilizes a hierarchical map-based route search algorithm. This algorithm searches the simulator's map metadata to find specific coordinates and paths that satisfy the logical requirements of the DSL (e.g., finding a T-junction where an NPC can approach from the right). Once the scenario is generated, the Scenario Monitor tracks the ADS's performance, logging rule violations, collisions, and timeouts.
Experimental Results
TARGET was evaluated using 54 rules from the Texas Driver Handbook across multiple simulators (CARLA, LGSVL, MetaDrive) and industrial-grade ADS stacks like Baidu Apollo 7.0 and Autoware.
Key findings include:
- High Parsing Accuracy: The GPT-4 powered parser achieved 91.8% accuracy in DSL conversion.
- Bug Discovery: Across 284 scenarios, TARGET uncovered 610 erroneous behaviors. For instance, Apollo 7.0 showed difficulties detecting slow-moving pedestrians, while Autoware suffered from localization errors leading to roadside collisions.
- Developer Confirmation: Several identified issues were confirmed by ADS developers, including a known limitation in perception functionality.
Conclusion
TARGET demonstrates that abstraction is a powerful tool for LLM-based testing. By using a high-level DSL as an intermediary, the framework leverages the LLM's reasoning strengths while delegating precise mathematical calculations to traditional algorithms. Future work will focus on handling complex sequential logic and incorporating multi-modal LLMs to interpret traffic rule diagrams alongside text.
Summary updated