Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test

Xuan Cai, Xuesong Bai, Zhiyong Cui, Danmu Xie, Daocheng Fu, Haiyang Yu, Yilong RenarXiv 2025 · DOI · arXiv

Introduction

Testing autonomous driving systems (ADS) is a critical step in ensuring road safety, yet it remains a significant bottleneck in the development lifecycle. Generating the vast array of "edge case" scenarios required for robust validation typically involves manual scripting using complex Domain Specific Languages (DSLs) like ASAM OpenScenario. This process is not only time-consuming—often requiring hundreds of lines of code for a single scenario—but also demands high technical expertise.

Text2Scenario (T2S) introduces a paradigm shift by leveraging Large Language Models (LLMs) to automate this process. T2S allows engineers to generate executable simulation scenarios directly from natural language descriptions, reducing generation time from over ten minutes to under a minute while maintaining high accuracy.

The Text2Scenario Framework

The T2S framework employs a multi-stage pipeline designed to bridge the gap between conceptual testing ideas and technical simulation files.

The Text2Scenario end-to-end pipeline, illustrating the flow from natural language input to simulation execution and evaluation.

1. Hierarchical Scenario Repository

To structure the world for the LLM, the authors deconstructed complex traffic environments into a six-layer hierarchy based on Safety of The Intended Functionality (SOTIF) tenets. This repository serves as a structured menu from which the model selects and configures components.

The six-tiered hierarchical scenario repository used to organize world elements into logical layers.

The layers include:

  • Road Topology: Intersections, roundabouts, and lane configurations.
  • Transportation Facilities: Road markers and traffic signs.
  • Temporary Changes: Road hazards like cones or warning signs.
  • Traffic Participants: Other vehicles and pedestrians.
  • Climate: Weather conditions and time of day.
  • Ego Vehicle: The vehicle under test and its planned behavior.

2. LLM-based Text Parsing

The core of the system is a sophisticated prompt engineering pipeline that converts text into a "logical scenario" representation. To overcome the stochastic nature of LLMs and minimize "hallucinations," the authors implemented:

  • Few-Shot Learning & Chain-of-Thought (CoT): Providing examples and forcing the model to reason through the scenario step-by-step to infer implicit logic (e.g., a "cut-in" implies the presence of a nearby vehicle).
  • Syntax Alignment Checking (SAC): Validating that the model's output aligns with the available components in the hierarchical repository.
  • Self-Consistency (SC): Running multiple iterations and using a majority vote to select the most stable and accurate representation.

3. Priority-based DSL Generation

Once a logical scenario is parsed, a specialized algorithm matches it against a library of DSL fragments. It uses a priority-based assembly process—for instance, establishing the road topology before placing vehicles—to ensure the resulting .xosc file is valid and executable within the Carla simulator.

Experimental Evaluation

The authors evaluated T2S using a benchmark of 92 unique scenario descriptions derived from NHTSA typologies, C-NCAP protocols, and expert-designed high-risk cases. These were expanded into 368 test scenarios by varying environmental factors.

Key Results

  • Accuracy: GPT-4 achieved a 92.3% average parsing accuracy, significantly outperforming GPT-3.5 (80.4%) and Yi-34B (88.1%).
  • Efficiency: T2S (using GPT-4) completed scenario construction in approximately 55 seconds, compared to over 600 seconds for human experts.
  • Safety Insights: The framework successfully identified 533 safety violations across four different AD stacks (Apollo, Autoware, Interfuser, and Dora-RS). Notably, the vision-only Dora-RS stack was found to be the most collision-prone, while the end-to-end Interfuser model exhibited the highest number of rule violations.

Conclusion

Text2Scenario demonstrates the potential of LLMs to streamline the safety testing of autonomous vehicles. By transforming natural language into executable code, it empowers testers to rapidly iterate on complex scenarios. Future work will focus on improving the system's robustness to input variations and expanding its ability to generate novel DSL code without relying on pre-defined libraries.

Summary updated

Abstract

Autonomous driving (AD) testing constitutes a critical methodology for assessing performance benchmarks prior to product deployment. The creation of segmented scenarios within a simulated environment is acknowledged as a robust and effective strategy; however, the process of tailoring these scenarios often necessitates laborious and time-consuming manual efforts, thereby hindering the development and implementation of AD technologies. In response to this challenge, we introduce Text2Scenario, a framework that leverages a Large Language Model (LLM) to autonomously generate simulation test scenarios that closely align with user specifications, derived from their natural language inputs. Specifically, an LLM, equipped with a meticulously engineered input prompt scheme functions as a text parser for test scenario descriptions, extracting from a hierarchically organized scenario repository the components that most accurately reflect the user's preferences. Subsequently, by exploiting the precedence of scenario components, the process involves sequentially matching and linking scenario representations within a Domain Specific Language corpus, ultimately fabricating executable test scenarios. The experimental results demonstrate that such prompt engineering can meticulously extract the nuanced details of scenario elements embedded within various descriptive formats, with the majority of generated scenarios aligning closely with the user's initial expectations, allowing for the efficient and precise evaluation of diverse AD stacks void of the labor-intensive need for manual scenario configuration. Project page: https://caixxuan.github.io/Text2Scenario.GitHub.io.
BibTeX
@misc{cai2025text,
  author = {Xuan Cai and Xuesong Bai and Zhiyong Cui and Danmu Xie and Daocheng Fu and Haiyang Yu and Yilong Ren},
  title = {{Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test}},
  year = {2025},
  eprint = {2503.02911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02911},
}
Figure