Assessing Modeling Variability in Autonomous Vehicle Accelerated Evaluation

Zhiyuan Huang, Mansur Arief, Henry Lam, Ding Zhao

Introduction

Statistically validating that autonomous vehicles (AVs) are safer than human drivers requires observing billions of miles of driving—a feat practically impossible through standard on-road testing. To address this, the industry employs "Accelerated Evaluation," utilizing Monte Carlo simulations and Importance Sampling to artificially increase the frequency of rare, safety-critical events like crashes.

However, these simulations rely on stochastic models fitted to finite real-world datasets. This introduces input uncertainty: the risk that model parameters are inaccurate due to limited data. Historically, AV safety evaluations have ignored this, reporting confidence intervals that only account for simulation variance. This oversight can lead to dangerous overconfidence in an AV's safety profile, as the reported intervals may fail to cover the true risk probability.

Methodology: Likelihood Ratio-Based Bootstrap

Quantifying input uncertainty typically requires the "classic bootstrap" method: resampling real-world data, refitting models, and re-running simulations hundreds or thousands of times. For computationally intensive AV simulations, this is prohibitively expensive.

The authors propose a Likelihood Ratio-Based Bootstrap to eliminate this bottleneck:

  1. Single Simulation Run: Execute the expensive Monte Carlo simulation once using initial estimated parameters and an Importance Sampling distribution.
  2. Parametric Resampling: Resample the real-world data to generate new bootstrap parameters.
  3. Likelihood Ratio Adjustment: Instead of re-simulating, the method reuses the original outputs. Results are adjusted by weighting them with the ratio of the new bootstrap distribution to the original distribution.

This approach decouples the simulation from the uncertainty analysis, allowing for input uncertainty calculation with zero additional simulation trials.

Experimental Setup

The method was evaluated using the Safety Pilot Model Deployment (SPMD) dataset, from which 12,304 lane-change scenarios were extracted. The study focused on the initial range and time-to-collision (TTC) for a frontal vehicle cutting into a lane at highway speeds (~30m/s) to fit the stochastic environment models.

Illustration of the lane-change scenario used for AV safety evaluation.

Key Insights and Results

A primary finding is that input uncertainty dominates simulation uncertainty. In the AV testing example, the confidence interval width for input uncertainty was approximately 1.5 times larger than that of simulation uncertainty.

Comparison of probability estimation with input and simulation uncertainty confidence intervals.

Crucially, while increasing simulation trials reduces simulation uncertainty toward zero, input uncertainty remains static. It is a bottleneck determined by the size of the original real-world dataset. You cannot simulate your way out of a lack of real-world data.

In a toy problem involving Gaussian tail probability, the proposed likelihood ratio method achieved ~94.4% coverage for a 95% confidence interval, matching the exact closed-form baseline. Conversely, ignoring input uncertainty resulted in coverage as low as 1.7%, highlighting the severe risk of standard evaluation methods.

Conclusion and Future Work

The Likelihood Ratio-Based Bootstrap provides a robust, computationally efficient way to bound AV safety risks. By accounting for the variability inherent in real-world data, researchers can avoid the overconfidence that arises from ignoring input uncertainty. While the parametric bootstrap provided the most reliable coverage for small datasets, future work will explore model-free analysis to broaden the technique's applicability to domains where parametric assumptions may not hold.

Summary updated

BibTeX
@inproceedings{huang2019assessing,
  author = {Zhiyuan Huang and Mansur Arief and Henry Lam and Ding Zhao},
  title = {{Assessing Modeling Variability in Autonomous Vehicle Accelerated Evaluation}},
  booktitle = {ITSC},
  year = {2019},
  doi = {10.1109/itsc.2019.8917406},
}
Figure