Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies

Shuo Feng, Yiheng Feng, Haowei Sun, Shao Bao, Aditi Misra, Yi Zhang, Henry X. LiuIEEE Trans. ITS 2019 · DOI · OpenReview

TL;DR

This paper presents practical implementations of a general methodology for generating testing scenarios for connected and automated vehicles (CAVs). By enhancing the methodology with reinforcement learning, they demonstrate significant acceleration of CAV evaluation across various scenarios, achieving comparable accuracy to traditional methods.

What Problem Does This Paper Address?

Developing and deploying safe and reliable CAVs hinges on rigorous testing. However, existing methods for generating testing scenarios often struggle with:

  • Limited Scenario Types: They can only handle simple, low-dimensional scenarios.
  • Restricted CAV Models: They are tailored to specific CAV models.
  • Narrow Performance Metrics: They primarily focus on safety, neglecting other crucial aspects like functionality.
  • The "Curse of Dimensionality": Searching for critical scenarios becomes computationally intractable as the number of variables (e.g., multiple surrounding vehicles, complex environmental conditions) increases.

This paper addresses these gaps by providing concrete case studies and enhancing the proposed methodology to handle high-dimensional scenarios, thereby enabling more comprehensive and efficient CAV evaluation.

Methods

The core of the paper builds upon the general Testing Scenario Library Generation (TSLG) methodology introduced in Part I. Key techniques include:

  • Scenario Criticality: A novel measure combining maneuver challenge and exposure frequency to prioritize scenarios for testing. This moves beyond simply focusing on worst-case scenarios.
  • Surrogate Model (SM) Construction: Using human driving data (Naturalistic Driving Data - NDD) to build models that represent generic driving behaviors, which are then used to estimate maneuver challenge.
  • Auxiliary Objective Function: Designed to guide the search for critical scenarios, tailored to specific performance metrics (safety or functionality).
  • Reinforcement Learning (RL) Enhancement: For high-dimensional scenarios (like car-following), a Temporal-Difference Reinforcement Learning (TD-RL) approach is employed to efficiently search the vast scenario space. This involves formulating the problem as a Markov Decision Process (MDP).
  • ε-greedy Sampling Policy: Used to sample testing scenarios from the generated library during CAV evaluation.

Datasets

  • Naturalistic Driving Data (NDD):
    • Safety Pilot Model Deployment (SPMD) program (University of Michigan): Used for the cut-in case study, providing data on longitudinal and lateral distances between vehicles.
  • Integrated Vehicle-Based Safety System (IVBSS) project: Used for the highway exit and car-following case studies, providing data on car-following events and trajectories.

The paper leverages these existing NDD datasets to estimate exposure frequency and calibrate surrogate models, rather than introducing new benchmarks.

Key Insights and Ideas

  • Bridging Theory and Practice: This paper moves the TSLG methodology from theoretical foundations to practical implementation with concrete case studies, offering valuable guidelines.
  • Functionality Matters: The highway exit case study highlights how the methodology can be adapted to evaluate functionality beyond just safety, by redefining the auxiliary objective function.
  • RL Tames High Dimensions: The car-following case demonstrates that RL is a powerful tool to overcome the "curse of dimensionality" in TSLG, making evaluation of complex scenarios feasible.
  • NDD is Key for Realism: Using NDD to calibrate SMs and estimate exposure frequency ensures that generated scenarios are representative of real-world driving conditions.
  • Efficiency Gains are Massive: The results consistently show dramatic reductions in testing time compared to traditional NDD-based evaluation, without sacrificing accuracy.

Key Results

The paper presents three case studies (cut-in, highway exit, car-following) demonstrating the effectiveness of the proposed TSLG methodology:

  • Cut-in Case (Safety): The proposed method achieved the required evaluation accuracy with 51 tests, while NDD evaluation required 9.63 × 10⁴ tests – a speedup of approximately 1,888 times.
  • Highway Exit Case (Functionality): The proposed method achieved the required estimation precision with 2.6 × 10³ tests, compared to 6.6 × 10⁵ tests for NDD evaluation – a speedup of approximately 255 times.
  • Car-Following Case (High-Dimensional Safety): The proposed method achieved the required estimation precision with 50 tests, while NDD evaluation required 1.875 × 10⁷ tests – a speedup of approximately 3.75 × 10⁵ times.

In all cases, the proposed method achieved the same evaluation accuracy as the NDD evaluation method.

Limitations and Future Work

  • Dissimilarity between SM and CAV: The efficiency of the proposed method is influenced by how well the surrogate model (often based on human driving) matches the actual CAV under test. Adaptive modification of SMs for different CAVs is suggested.
  • NDD Requirement: Large-scale NDD is crucial for accurate exposure frequency estimation, which can be a limitation if such data is not readily available. However, the authors note that data collection is becoming more accessible.
  • Extremely High Dimensions: While RL helps with high dimensions, scenarios with hundreds of background vehicles present further challenges that may require improvements to the RL-enhanced method.
  • Adaptive Library Updates: Investigating methods to adaptively update the SM and the generated library using data collected during the testing process is a promising future direction.

Summary updated

Abstract

Testing scenario library generation (TSLG) is a critical step for the development and deployment of connected and automated vehicles (CAVs). In Part I of this study, a general methodology for TSLG is proposed, and theoretical properties are investigated regarding the accuracy and efficiency of CAV evaluation. This paper aims to provide implementation examples and guidelines, and to enhance the proposed methodology under high-dimensional scenarios. Three typical cases, including cut-in, highway-exit, and car-following, are designed and studied in this paper. For each case, the process of library generation and CAV evaluation is elaborated. To address the challenges brought by high dimensions, the proposed methodology is further enhanced by reinforcement learning technique. For all three cases, results show that the proposed methods can accelerate the CAV evaluation process by multiple magnitudes with same evaluation accuracy, if compared with the on-road test method.
BibTeX
@article{feng2019testing,
  author = {Shuo Feng and Yiheng Feng and Haowei Sun and Shao Bao and Aditi Misra and Yi Zhang and Henry X. Liu},
  title = {{Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2019},
  doi = {10.1109/tits.2020.2988309},
}
Figure