Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework

Shuo Feng, Yiheng Feng, Haowei Sun, Yi Zhang, Henry X. LiuIEEE Trans. ITS 2022 · OpenReview

TL;DR

This paper introduces an Adaptive Testing Scenario Library Generation (ATSLG) framework to create customized testing libraries for Connected and Automated Vehicles (CAVs). By adaptively compensating for performance differences between a surrogate model and the actual CAV, ATSLG significantly accelerates the evaluation process, achieving speedups of 10-100 times compared to traditional methods.

What Problem Does This Paper Address?

Evaluating the safety and performance of CAVs requires comprehensive testing across a vast number of potential scenarios. Generating a representative and efficient testing scenario library is a major challenge. Existing methods often rely on surrogate models (SMs) to estimate scenario criticality without knowing the exact CAV under test. However, performance discrepancies between the SM and the actual CAV can lead to suboptimal libraries, meaning either critical scenarios are missed (underweight) or non-critical ones are included (overweight). This inefficiency increases the number of tests needed to achieve a desired level of confidence in the evaluation results. The paper aims to address this by developing an adaptive framework that customizes the testing library for a specific CAV, thereby reducing suboptimal scenarios and accelerating the evaluation.

Methods

The core of the proposed ATSLG method is a Bayesian optimization scheme. This approach iteratively refines a testing scenario library by:

  • Estimating Dissimilarities: A classification-based Gaussian Process Regression (GPR) is used to estimate the non-stationary performance dissimilarities between the surrogate model and the actual CAV. This method handles scenarios where the dissimilarity function is not uniform across all possible driving situations.
  • Adaptive Scenario Selection: A novel acquisition function is designed to intelligently select the next most informative scenario to test. This function balances exploiting known information about the dissimilarity and exploring uncertain regions to maximize the reduction in estimation variance.
  • Surrogate Model and Library Update: Based on the observed dissimilarity from testing, the surrogate model is updated, and the testing scenario library is refined. This iterative process progressively improves the library's accuracy and relevance to the specific CAV.
  • Customized Library Generation: The adaptive process aims to create an "optimal library" that closely matches the CAV's performance, minimizing both underweight and overweight scenarios.

Datasets

  • Naturalistic Driving Data (NDD): Used to estimate the exposure frequency of cut-in scenarios from the Safety Pilot Model Deployment program at the University of Michigan. This provided a dataset of 414,770 qualified cut-in events.
  • Not explicitly mentioned: The paper focuses on a simulated cut-in case study, implying that the "ground truth" for dissimilarity was either generated through a more detailed simulation or derived from prior knowledge.

Key Insights and Ideas

  • Performance Dissimilarities are Key: The paper highlights that the mismatch between surrogate models and actual CAVs is a primary source of inefficiency in testing. Addressing this directly is crucial for accelerating evaluation.
  • Bayesian Optimization for Efficiency: Applying Bayesian optimization is a powerful strategy for efficiently exploring complex, expensive-to-evaluate objective functions (like CAV testing outcomes).
  • Adaptive Customization is Superior: A one-size-fits-all, pre-determined library is less efficient than a library that adapts to the specific characteristics of the CAV being tested.
  • Non-Stationary GPR for Realistic Dissimilarities: Using classification-based GPR allows for modeling dissimilarity functions that vary across different driving conditions, which is more realistic than assuming stationarity.
  • Informativeness-Driven Scenario Selection: The novel acquisition function is designed to actively seek out scenarios that provide the most information for reducing uncertainty, leading to faster convergence.

Key Results

  • Significant Speedup: The ATSLG method demonstrated a substantial acceleration in the evaluation process. For a cut-in case study, it achieved speedups of 1570 times and 17 times compared to on-road testing and offline library evaluation, respectively, to reach a specific precision level.
  • Reduced Test Requirements: To achieve a relative half-width of 0.2, the ATSLG method required approximately 121 tests, compared to 1.9 x 10^5 for offline library evaluation.
  • Improved Library Quality: The customized library generated by ATSLG was shown to be significantly improved compared to the offline generated library, with fewer suboptimal scenarios.
  • Effective Dissimilarity Compensation: The adaptive process successfully reduced performance dissimilarities between the SM and the CAV, approaching zero after 50 iterations.

Limitations and Future Work

  • High-Dimensional Scenarios: The paper acknowledges that extending the ATSLG problem to high-dimensional scenarios presents a significant challenge.
  • Realistic Testing Platforms: The authors suggest applying the proposed method to more realistic CAV testing platforms with pre-established scenario libraries.
  • Exploration of Other Metrics: While the paper focuses on safety evaluation (accident rate), future work could explore applying ATSLG to other performance metrics.
  • Covariance Function Design: The paper notes that the chosen covariance function for GPR is not unique or necessarily optimal, suggesting further investigation into better covariance function designs.

Summary updated

Abstract

How to generate testing scenario libraries for connected and automated vehicles (CAVs) is a major challenge faced by the industry. In previous studies, to evaluate maneuver challenge of a scenario, surrogate models (SMs) are often used without explicit knowledge of the CAV under test. However, performance dissimilarities between the SM and the CAV under test usually exist, and it can lead to the generation of suboptimal scenario libraries. In this article, an adaptive testing scenario library generation (ATSLG) method is proposed to solve this problem. A customized testing scenario library for a specific CAV model is generated through an adaptive process. To compensate for the performance dissimilarities and leverage each test of the CAV, Bayesian optimization techniques are applied with classification-based Gaussian Process Regression and a newly designed acquisition function. Comparing with a pre-determined library, a CAV can be tested and evaluated in a more efficient manner with the customized library. To validate the proposed method, a cut-in case study is investigated and the results demonstrate that the proposed method can further accelerate the evaluation process by a few orders of magnitude.
BibTeX
@article{feng2022testing,
  author = {Shuo Feng and Yiheng Feng and Haowei Sun and Yi Zhang and Henry X. Liu},
  title = {{Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=q30tH77qSw},
}
Figure