Abstract
Verification and validation are major challenges for developing automated driving systems. A concept that gets more and more recognized for testing in automated driving is scenario-based testing. However, it introduces the problem of what scenarios are relevant for testing and which are not. This work aims to find relevant, interesting, or critical parameter sets within logical scenarios by utilizing Bayes optimization and Gaussian processes. The parameter optimization is done by comparing and evaluating six different metrics in two urban intersection scenarios. Finally, a list of ideas this work leads to and should be investigated further is presented.
Introduction
The validation of Automated Driving Systems (ADS) represents one of the most significant hurdles in the path toward full autonomy. While simulation offers a safe and scalable alternative to real-world testing, it introduces the "curse of dimensionality." In a typical urban intersection, the combination of vehicle speeds, pedestrian timings, and environmental factors creates a nearly infinite space of possible concrete scenarios.
Brute-force testing is computationally prohibitive; for instance, a single logical scenario with three parameters and 50 variations each results in 125,000 test cases. This paper addresses the need for an intelligent exploration strategy to identify critical edge cases—where collisions or near-misses occur—without wasting resources on thousands of mundane, safe simulations.
Methodology: Directed Scenario Exploration
The researchers proposed a Directed Scenario Exploration framework that replaces random sampling with Bayesian Optimization (BO) supported by Gaussian Processes (GP).
The Optimization Process
BO treats the simulation environment as a "black box." It builds a statistical surrogate model of how "critical" a scenario is based on previous results. The framework then employs an acquisition function (specifically Thompson sampling) to decide which parameter set to test next. This approach balances the need to explore unknown regions of the parameter space with the need to exploit areas already known to be dangerous.
Simulation Stack and Metrics
The core simulation was conducted using IPG CarMaker, integrated with a modular driving function running on the Robot Operating System (ROS). To quantify what constitutes a "critical" scenario, the authors evaluated five distinct metrics:
- Euclidean Distance: Simple spatial distance between actors.
- Trajectory Distance: Distance measured along the predicted paths of the actors.
- Worst-Time-To-Collision (WTTC): A variation of TTC that accounts for non-car-following geometries.
- Gap Time (GT): The predicted time interval between actors crossing the same spatial point.
- Post-Encroachment-Time (PET): The actual time difference between one actor leaving a location and another entering it.
Experimental Setup
The study utilized two logical scenarios at urban intersections to generate data:
- Scenario A: The ego vehicle turns right, crossing the paths of a pedestrian and a truck.
- Scenario B: The ego vehicle turns left, crossing the paths of a pedestrian, a truck, and an oncoming car.
The combined parameter space for these scenarios totaled 625,000 possible concrete variations.
Key Insights and Results
Efficiency and Search Reduction
The BO approach successfully identified critical clusters by running only ~430 simulations out of the potential 625,000. This represents a massive reduction in computational overhead, finding critical scenarios using roughly 0.07% of the total possible simulations.
Parameter Sensitivity and "Dead Parameters"
In Scenario A, the researchers discovered that the speed of the "other" car had almost no impact on the criticality of the pedestrian interaction. Identifying these "dead parameters" is vital for engineers, as it allows them to prune the parameter space and focus on variables that actually drive risk, such as pedestrian delay and the ego vehicle's starting position.
Metric Sensitivity
While different metrics (like PET vs. WTTC) generally agreed on the location of "danger zones," they varied in sensitivity. Distance metrics provided a smoother gradient for the optimizer to follow, whereas PET only became "critical" when actors were very close in time. This suggests that the choice of metric can significantly influence the speed of the optimization convergence.
Critical Clusters
The study confirmed that criticality is not random; it exists in "clusters" within the parameter space. Once the algorithm identifies the boundary of a cluster (e.g., a specific pedestrian delay that causes a near-miss), it can efficiently map out the entire failure region.
Limitations and Future Work
The current optimization focused primarily on pairwise interactions (e.g., ego vehicle vs. a single pedestrian). However, in complex urban traffic, criticality is often the result of the cumulative behavior of all actors.
Future research directions include:
- Objective Scene Metrics: Moving toward metrics that evaluate the entire scene holistically rather than just pairwise interactions.
- Cross-Tool Validation: Testing the framework across different simulation tools to ensure the findings are not specific to the IPG CarMaker/ROS stack.
- Severity Comparison: Developing methods to better compare the relative severity of different metrics, as a 1.5s PET may not represent the same risk level as a 2-meter Euclidean distance in different contexts.
Summary updated