Scenario Generation for Autonomous Vehicles with Deep-Learning-Based Heterogeneous Driver Models: Implementation and Verification

Li Gao, Rui Zhou, Kai Zhang

TL;DR

This paper introduces a novel deep-learning-based approach to generate more realistic and challenging scenarios for testing autonomous vehicles (AVs). By modeling heterogeneous driver behaviors (aggressive, normal, conservative), they demonstrate an increase in dangerous scenarios, accelerating AV testing and improving safety verification.

What Problem Does This Paper Address?

Testing autonomous vehicles (AVs) in real-world conditions is expensive and time-consuming, especially for rare, hazardous events. Traditional virtual testing often relies on fixed scenario spaces, failing to capture the variability of human drivers. In reality, different drivers exhibit distinct behaviors (e.g., aggressive vs. conservative) even in the same situation. This heterogeneity is crucial for generating realistic and challenging scenarios that can effectively stress-test AV systems and uncover potential safety flaws. The paper aims to bridge this gap by creating a dynamic scenario generation method that accounts for driver diversity.

Methods

The core of this work is a two-stage deep-learning driver model designed to capture heterogeneous driving styles.

  1. Maneuver Module (MM): This module uses a Long Short-Term Memory (LSTM) encoder to predict the probability of different maneuvers (e.g., lane changes, braking) based on historical scene context.
  2. Action Module (AM): This module takes the predicted maneuver and context to generate a distribution of possible future endpoints. An endpoint is then sampled as a driver's intention, which, along with historical trajectory, is used by a Multilayer Perceptron (MLP) to predict the next frame's acceleration.

This model is trained to minimize a combined loss function that includes focal loss for maneuver probability (to handle data imbalance), negative log-likelihood for endpoint distribution, and mean squared error for acceleration. Heterogeneous driver models are created by training separate models for aggressive, normal, and conservative driving styles.

For scenario generation, the AV is simulated using the Intelligent Driver Model (IDM), while surrounding vehicles (SVs) are controlled by these heterogeneous driver models. The process involves initializing scenarios, simulating interactions, calculating safety indicators (like Time-to-Collision - TTC), and marking scenarios as dangerous if critical thresholds are met.

Datasets

The primary dataset used for training the driver models is the highD dataset. This dataset contains UAV-recorded trajectory data from over 110,500 vehicles on German highways, sampled at 25 Hz. The data was down-sampled to 5 Hz for training. The authors used k-means clustering on features like mean, variance, and maximum values of velocity and acceleration to classify drivers into aggressive, normal, and conservative categories.

Key Insights and Ideas

  • Driver Heterogeneity is Key: The most significant insight is that simply simulating with a single "average" driver model is insufficient. Accounting for diverse driving styles (aggressive, conservative) dramatically impacts the types and frequency of dangerous scenarios generated.
  • Controlling Aggression for Danger: By strategically adjusting the number and placement of aggressive drivers, the authors can significantly increase the proportion of challenging and dangerous scenarios in the test space. This is a direct and controllable way to create more effective test cases.
  • Accelerated Testing: The ability to generate more dangerous scenarios more efficiently directly translates to accelerating the testing process for AVs. This means AV developers can find critical bugs and safety issues faster.
  • Realistic Scenario Generation: The deep-learning approach, particularly the two-stage maneuver-action model, allows for more nuanced and realistic prediction of driver behavior, leading to more plausible and complex scenarios than traditional sampling methods.

Key Results

  • Increased Dangerous Scenarios: Experiments showed that using heterogeneous driver models (specifically, all SVs as aggressive) led to a significantly higher percentage of challenging scenarios compared to using a single, original driver model. For instance, in Initial Scenario 1, the percentage of challenging scenarios increased from 7% (Original Driver Model) to 13% (All SVs Aggressive).
  • Model Performance: The proposed driver models achieved good performance, with Negative Log Likelihood (NLL) values comparable to or better than a baseline CS-LSTM model, and very low cross-entropy for maneuver probability (e.g., 0.018 for the aggressive model), indicating accurate maneuver prediction.
  • Driving Strategy Verification: The generated scenarios were used to evaluate two driving strategies: a defensive RSS (Responsibility-Sensitive Safety) model and a more efficient "negotiated" strategy. Both strategies successfully navigated dangerous car-following and cut-in scenarios, demonstrating the validity and usefulness of the generated scenarios for evaluating AV safety and efficiency. The negotiated strategy showed higher traffic efficiency while RSS guaranteed absolute security.

Limitations and Future Work

  • Driver Style Granularity: The authors acknowledge that driver styles could be delineated in more detail. Future work could explore finer-grained driver typologies or consider heterogeneity from other perspectives beyond just aggressive/normal/conservative.
  • Complexity and Danger: As driver models become more diverse and scenarios more complex, the potential for danger increases. Future research could focus on how to systematically explore these increasingly complex and dangerous scenarios.
  • Real-World Validation: While the paper focuses on simulation, future work could involve validating the generated scenarios and the effectiveness of the driver models against real-world driving data and incidents.

Summary updated

Abstract

Virtual testing requires hazardous scenarios to effectively test autonomous vehicles (AVs). Existing studies have obtained rarer events by sampling methods in a fixed scenario space. In reality, heterogeneous drivers behave differently when facing the same situation. To generate more realistic and efficient scenarios, we propose a two-stage heterogeneous driver model to change the number of dangerous scenarios in the scenario space. We trained the driver model using the HighD dataset, and generated scenarios through simulation. Simulations were conducted in 20 experimental groups with heterogeneous driver models and 5 control groups with the original driver model. The results show that, by adjusting the number and position of aggressive drivers, the percentage of dangerous scenarios was significantly higher compared to that of models not accounting for driver heterogeneity. To further verify the effectiveness of our method, we evaluated two driving strategies: car-following and cut-in scenarios. The results verify the effectiveness of our approach. Cumulatively, the results indicate that our approach could accelerate the testing of AVs.
BibTeX
@inproceedings{gao2023scenario,
  author = {Li Gao and Rui Zhou and Kai Zhang},
  title = {{Scenario Generation for Autonomous Vehicles with Deep-Learning-Based Heterogeneous Driver Models: Implementation and Verification}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=chkniMMTpQ},
}
Figure