Introduction
Testing autonomous driving systems (ADS) is a critical step in ensuring road safety, yet it remains a significant bottleneck in the development lifecycle. Generating the vast array of "edge case" scenarios required for robust validation typically involves manual scripting using complex Domain Specific Languages (DSLs) like ASAM OpenScenario. This process is not only time-consuming—often requiring hundreds of lines of code for a single scenario—but also demands high technical expertise.
Text2Scenario (T2S) introduces a paradigm shift by leveraging Large Language Models (LLMs) to automate this process. T2S allows engineers to generate executable simulation scenarios directly from natural language descriptions, reducing generation time from over ten minutes to under a minute while maintaining high accuracy.
The Text2Scenario Framework
The T2S framework employs a multi-stage pipeline designed to bridge the gap between conceptual testing ideas and technical simulation files.
1. Hierarchical Scenario Repository
To structure the world for the LLM, the authors deconstructed complex traffic environments into a six-layer hierarchy based on Safety of The Intended Functionality (SOTIF) tenets. This repository serves as a structured menu from which the model selects and configures components.
The layers include:
- Road Topology: Intersections, roundabouts, and lane configurations.
- Transportation Facilities: Road markers and traffic signs.
- Temporary Changes: Road hazards like cones or warning signs.
- Traffic Participants: Other vehicles and pedestrians.
- Climate: Weather conditions and time of day.
- Ego Vehicle: The vehicle under test and its planned behavior.
2. LLM-based Text Parsing
The core of the system is a sophisticated prompt engineering pipeline that converts text into a "logical scenario" representation. To overcome the stochastic nature of LLMs and minimize "hallucinations," the authors implemented:
- Few-Shot Learning & Chain-of-Thought (CoT): Providing examples and forcing the model to reason through the scenario step-by-step to infer implicit logic (e.g., a "cut-in" implies the presence of a nearby vehicle).
- Syntax Alignment Checking (SAC): Validating that the model's output aligns with the available components in the hierarchical repository.
- Self-Consistency (SC): Running multiple iterations and using a majority vote to select the most stable and accurate representation.
3. Priority-based DSL Generation
Once a logical scenario is parsed, a specialized algorithm matches it against a library of DSL fragments. It uses a priority-based assembly process—for instance, establishing the road topology before placing vehicles—to ensure the resulting .xosc file is valid and executable within the Carla simulator.
Experimental Evaluation
The authors evaluated T2S using a benchmark of 92 unique scenario descriptions derived from NHTSA typologies, C-NCAP protocols, and expert-designed high-risk cases. These were expanded into 368 test scenarios by varying environmental factors.
Key Results
- Accuracy: GPT-4 achieved a 92.3% average parsing accuracy, significantly outperforming GPT-3.5 (80.4%) and Yi-34B (88.1%).
- Efficiency: T2S (using GPT-4) completed scenario construction in approximately 55 seconds, compared to over 600 seconds for human experts.
- Safety Insights: The framework successfully identified 533 safety violations across four different AD stacks (Apollo, Autoware, Interfuser, and Dora-RS). Notably, the vision-only Dora-RS stack was found to be the most collision-prone, while the end-to-end Interfuser model exhibited the highest number of rule violations.
Conclusion
Text2Scenario demonstrates the potential of LLMs to streamline the safety testing of autonomous vehicles. By transforming natural language into executable code, it empowers testers to rapidly iterate on complex scenarios. Future work will focus on improving the system's robustness to input variations and expanding its ability to generate novel DSL code without relying on pre-defined libraries.
Summary updated