TL;DR
This paper introduces an Adaptive Testing Scenario Library Generation (ATSLG) framework to create customized testing libraries for Connected and Automated Vehicles (CAVs). By adaptively compensating for performance differences between a surrogate model and the actual CAV, ATSLG significantly accelerates the evaluation process, achieving speedups of 10-100 times compared to traditional methods.
What Problem Does This Paper Address?
Evaluating the safety and performance of CAVs requires comprehensive testing across a vast number of potential scenarios. Generating a representative and efficient testing scenario library is a major challenge. Existing methods often rely on surrogate models (SMs) to estimate scenario criticality without knowing the exact CAV under test. However, performance discrepancies between the SM and the actual CAV can lead to suboptimal libraries, meaning either critical scenarios are missed (underweight) or non-critical ones are included (overweight). This inefficiency increases the number of tests needed to achieve a desired level of confidence in the evaluation results. The paper aims to address this by developing an adaptive framework that customizes the testing library for a specific CAV, thereby reducing suboptimal scenarios and accelerating the evaluation.
Methods
The core of the proposed ATSLG method is a Bayesian optimization scheme. This approach iteratively refines a testing scenario library by:
- Estimating Dissimilarities: A classification-based Gaussian Process Regression (GPR) is used to estimate the non-stationary performance dissimilarities between the surrogate model and the actual CAV. This method handles scenarios where the dissimilarity function is not uniform across all possible driving situations.
- Adaptive Scenario Selection: A novel acquisition function is designed to intelligently select the next most informative scenario to test. This function balances exploiting known information about the dissimilarity and exploring uncertain regions to maximize the reduction in estimation variance.
- Surrogate Model and Library Update: Based on the observed dissimilarity from testing, the surrogate model is updated, and the testing scenario library is refined. This iterative process progressively improves the library's accuracy and relevance to the specific CAV.
- Customized Library Generation: The adaptive process aims to create an "optimal library" that closely matches the CAV's performance, minimizing both underweight and overweight scenarios.
Datasets
- Naturalistic Driving Data (NDD): Used to estimate the exposure frequency of cut-in scenarios from the Safety Pilot Model Deployment program at the University of Michigan. This provided a dataset of 414,770 qualified cut-in events.
- Not explicitly mentioned: The paper focuses on a simulated cut-in case study, implying that the "ground truth" for dissimilarity was either generated through a more detailed simulation or derived from prior knowledge.
Key Insights and Ideas
- Performance Dissimilarities are Key: The paper highlights that the mismatch between surrogate models and actual CAVs is a primary source of inefficiency in testing. Addressing this directly is crucial for accelerating evaluation.
- Bayesian Optimization for Efficiency: Applying Bayesian optimization is a powerful strategy for efficiently exploring complex, expensive-to-evaluate objective functions (like CAV testing outcomes).
- Adaptive Customization is Superior: A one-size-fits-all, pre-determined library is less efficient than a library that adapts to the specific characteristics of the CAV being tested.
- Non-Stationary GPR for Realistic Dissimilarities: Using classification-based GPR allows for modeling dissimilarity functions that vary across different driving conditions, which is more realistic than assuming stationarity.
- Informativeness-Driven Scenario Selection: The novel acquisition function is designed to actively seek out scenarios that provide the most information for reducing uncertainty, leading to faster convergence.
Key Results
- Significant Speedup: The ATSLG method demonstrated a substantial acceleration in the evaluation process. For a cut-in case study, it achieved speedups of 1570 times and 17 times compared to on-road testing and offline library evaluation, respectively, to reach a specific precision level.
- Reduced Test Requirements: To achieve a relative half-width of 0.2, the ATSLG method required approximately 121 tests, compared to 1.9 x 10^5 for offline library evaluation.
- Improved Library Quality: The customized library generated by ATSLG was shown to be significantly improved compared to the offline generated library, with fewer suboptimal scenarios.
- Effective Dissimilarity Compensation: The adaptive process successfully reduced performance dissimilarities between the SM and the CAV, approaching zero after 50 iterations.
Limitations and Future Work
- High-Dimensional Scenarios: The paper acknowledges that extending the ATSLG problem to high-dimensional scenarios presents a significant challenge.
- Realistic Testing Platforms: The authors suggest applying the proposed method to more realistic CAV testing platforms with pre-established scenario libraries.
- Exploration of Other Metrics: While the paper focuses on safety evaluation (accident rate), future work could explore applying ATSLG to other performance metrics.
- Covariance Function Design: The paper notes that the chosen covariance function for GPR is not unique or necessarily optimal, suggesting further investigation into better covariance function designs.
Summary updated