Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

Xingyu Zhao, Valentin Robu, David Flynn, Kizito Salako, Lorenzo Strigini

This paper, published by researchers from Heriot-Watt University and City, University of London, tackles one of the most daunting hurdles in the autonomous vehicle (AV) industry: proving that a self-driving car is actually safer than a human driver.

TL;DR

Road testing alone requires billions of miles to statistically prove AV safety, which is practically impossible. This paper proposes a Conservative Bayesian Inference (CBI) framework that combines real-world testing with prior knowledge (like safety audits) to reach confidence faster, while using Software Reliability Growth Models (SRGMs) to track how AVs improve over time.

What Problem Does This Paper Address?

The "RAND challenge" is well-known in the industry: to prove an AV is 20% safer than a human driver with 95% confidence, a fleet would need to drive roughly 5 billion miles without a fatality. At current testing rates, this would take hundreds of years.

The authors identify two major gaps in current safety assessments:

  1. Ignoring Prior Knowledge: Standard "classical" statistics treat every test as if we know nothing about the car's design. In reality, engineers have results from simulations, formal verification, and safety monitors.
  2. The "Static" Assumption: Most models assume the AV's reliability is constant. However, AV software is constantly updated to fix bugs, meaning its reliability should (ideally) improve over time.

Methods

The paper introduces two distinct statistical approaches to bridge these gaps:

  • Conservative Bayesian Inference (CBI): Unlike standard Bayesian methods that require a complete (and often arbitrary) "prior" probability distribution, CBI works with partial prior knowledge. For example, an assessor might say, "I am 80% confident the failure rate is below $10^{-3}$." CBI then identifies the most "pessimistic" distribution that fits this claim, ensuring safety claims aren't built on over-optimistic assumptions.
  • Software Reliability Growth Models (SRGMs): These are mathematical models used to predict future failures based on the history of "disengagements" (when a human has to take over).
  • Recalibration: Since no single SRGM is perfect, the authors use a recalibration technique that looks at how a model's past predictions performed and adjusts future forecasts to be more accurate.

Datasets

The authors evaluate their methods using:

  • Waymo Public Road Testing Data: 51 months of data (up to 2018) including 528 disengagements.
  • RAND Study Benchmarks: Used as a baseline for "classical" statistical requirements for fatality-free miles.

Key Insights and Ideas

  • Prior Knowledge is a Force Multiplier: If you have strong evidence from non-road testing (e.g., a verified safety "watchdog" system), the number of miles required to prove a safety claim drops drastically.
  • The "Fatality Compensation" Math: The paper provides a way to calculate exactly how many extra miles a company must drive to "make up" for a single newly observed fatality to maintain their safety claim.
  • Disengagements $\neq$ Fatalities: While disengagements ($dpm$) are a useful proxy for software maturity, the authors warn that they shouldn't be used as a direct measure of fatality risk ($pfm$). A car might have fewer disengagements but still be prone to rare, catastrophic "edge case" failures.
  • Recalibration is Mandatory: In the Waymo dataset, different models (like Goel-Okumoto vs. Littlewood-Verrall) disagreed wildly. Recalibration significantly reduced this disagreement, making the models a viable tool for test planning.

Key Results

  • Miles Saved: To claim an AV is as safe as a human ($1.09 \times 10^{-8}$ fatalities per mile) with 95% confidence:
  • Classical (RAND) approach: Needs 275 million fatality-free miles.
  • CBI with a strong prior (90% confidence): Needs only 69 million fatality-free miles.
  • Waymo's Growth: Applying SRGMs to Waymo's data showed a clear reliability growth trend. By the end of the 51-month period, the best estimate for the "Median Miles to next Disengagement" (MMTD) was approximately 7,000–8,000 miles.
  • The CBI "Ceiling": The authors proved that there is a mathematical ceiling on how many extra miles are needed to compensate for a failure, regardless of how high your confidence level is.

Limitations and Future Work

  • The "Prior" Problem: While CBI uses partial prior knowledge, the authors admit that justifying those initial "prior beliefs" (e.g., "I'm 80% sure...") remains an unsolved challenge in safety engineering.
  • Stratified Testing: The current models treat all miles as equal. Future work needs to account for the fact that 1,000 miles in snowy Manhattan are much more informative than 1,000 miles on a sunny Arizona highway.
  • Human Factors: The models don't yet account for the "safety driver" effect—how a human's presence and trust in the system change the frequency of disengagements.

Summary updated

Abstract

There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents, road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing - knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo's public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracy and then improve forecasts. We show that, coupled with accuracy assessment and recalibration techniques, SRGMs could be a valuable test planning aid.
BibTeX
@inproceedings{zhao2019assessing,
  author = {Xingyu Zhao and Valentin Robu and David Flynn and Kizito Salako and Lorenzo Strigini},
  title = {{Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing}},
  booktitle = {ISSRE},
  year = {2019},
  url = {https://openreview.net/forum?id=sm9vekYNibW},
}

Loading related content...

Figure