TL;DR
This paper introduces AutoSceneGen, a novel framework that leverages in-context learning (ICL) from Large Language Models (LLMs) to automatically generate diverse and safety-critical traffic scenarios for training autonomous vehicle (AV) motion planners. By translating natural language descriptions into simulator-executable scripts, AutoSceneGen significantly reduces the human effort and cost associated with creating realistic and robust AV testing datasets.
What Problem Does This Paper Address?
Developing safe and reliable autonomous vehicles (AVs) requires extensive testing under a wide range of conditions. A critical challenge is the generation of diverse and realistic traffic scenarios, especially "safety-critical" ones that are rare but crucial for ensuring AV robustness. Traditional methods for creating these scenarios are either expensive (manual annotation of real-world data) or insufficient (existing datasets may not capture the full spectrum of complex interactions). Simulators like CARLA can generate scenarios, but manually scripting them is time-consuming and labor-intensive. Existing automated methods often struggle with low-resource scenarios or require significant post-processing. This paper addresses the need for a cost-effective, scalable, and automated approach to generate rich, safety-critical traffic scenarios that can improve the performance of AV motion planners.
Methods
The core of AutoSceneGen lies in its innovative use of In-Context Learning (ICL) with Large Language Models (LLMs). Instead of training or fine-tuning specialized models for scenario generation, AutoSceneGen leverages the LLM's ability to understand and generate text based on provided examples within the prompt itself.
Here's a breakdown of the process:
- Scenario Description Input: Users provide natural language descriptions of desired traffic scenarios (e.g., "In downtown area, during a drizzly noon, there are vehicles malfunctioning windshield wipers...").
- Exception Handling & Filtering: An "Exception Handler" filters out adversarial or irrelevant inputs. A "Filter" then processes the description, translating simulator-incompatible terms into those compatible with the simulator's APIs (e.g., replacing "storm" with "rain" for CARLA).
- ICL Prompt Construction: The filtered description is combined with pre-constructed ICL exemplars. These exemplars can be zero-shot, one-shot, or few-shot, depending on the LLM's familiarity with the simulator's APIs and the scenario's complexity.
- LLM Scenario Generation: The LLM processes the prompt and generates scenario configurations, often including explanations.
- Validation: A "Validator" checks each generated API call for compatibility with the simulator. It replaces unsupported terms with synonyms or ignores them. It also cleans up the output to ensure only executable code remains.
- Simulation: The validated configuration file is used to run the scenario in a simulator (like CARLA).
This approach bypasses the need for extensive manual scripting or model training, making scenario generation more efficient and adaptable.
Datasets
The paper evaluates its generated datasets using existing benchmarks and also uses its own generated data for training and evaluation.
- Existing Datasets: ApolloScapes, NGSIM, ETH/UCY, VIRAT/ActEV.
- AutoSceneGen's Contribution: The paper generates a new dataset of traffic scenarios using its framework. This dataset is characterized by its dynamic generation based on user prompts and its focus on heterogeneity and safety-critical scenarios. The paper highlights that while ApolloScapes has a static volume of data, AutoSceneGen's values are dynamically calculated.
Key Insights and Ideas
- LLMs as Scenario Engineers: The paper demonstrates that LLMs, through ICL, can be powerful tools for translating high-level, human-readable scenario descriptions into precise, simulator-executable code, significantly democratizing scenario generation.
- Cost-Effectiveness and Scalability: AutoSceneGen offers a more cost-effective and scalable alternative to manual scenario creation or reliance on limited real-world datasets, especially for rare and safety-critical events.
- Bridging the Gap in Simulation: The framework effectively bridges the gap between abstract scenario descriptions and concrete simulation inputs, a long-standing challenge in AV testing.
- ICL's Practical Application: The work showcases a practical and effective application of ICL beyond traditional NLP tasks, extending it to complex domain-specific code generation for simulation.
Key Results
The primary evaluation focuses on the impact of AutoSceneGen's generated data on motion planner performance.
- Improved Motion Planner Performance: Motion planners trained on datasets augmented with AutoSceneGen data significantly outperformed those trained solely on real-world data (e.g., ApolloScapes).
- Quantitative Improvements: In experiments using the TrafficPredict method, combining AutoSceneGen data with ApolloScapes led to substantial reductions in Average Displacement Error (ADE) and Final Displacement Error (FDE) for vehicles, bicycles, and pedestrians. For instance, using "Ours" (AutoSceneGen data) with TrafficPredict achieved ADE of 0.033 and FDE of 0.020 for vehicles, a significant improvement over ApolloScapes alone (ADE 0.085, FDE 0.091).
- Efficiency: The paper claims AutoSceneGen is more cost-effective and efficient than collecting real-world datasets. Table 1 shows acquisition times, with AutoSceneGen's dynamic calculation of values based on speed per minute.
- Scenario Diversity: The generated datasets are described as logically realistic and heterogeneous, capturing a wider range of scenarios than some existing datasets.
Limitations and Future Work
- Simulator Dependency: The framework's effectiveness is tied to the capabilities of both the LLM and the chosen simulator. If the LLM lacks strong ICL or the simulator cannot handle complex scenarios, generating corner cases becomes difficult.
- Explainability: The explainability of the generated scenario logic is limited by the LLM's inherent explainability. While detailed configuration files are stored, tracing the rationale can still be challenging.
- Data Availability and Rights: Obtaining sufficient ICL examples and ensuring usage rights for simulator documentation can be a hurdle.
- Future Work: The authors suggest developers may need to create custom virtual environments or choose different simulators based on their regulatory requirements. They also note that the framework's utility depends on the LLM's ICL capabilities and the simulator's complexity handling.
Summary updated