A behavior driven approach for sampling rare event situations for autonomous vehicles

Atrisha Sarkar, Krzysztof Czarnecki

Here is a summary of the paper tailored for a technically literate audience.

TL;DR

Evaluating autonomous vehicles (AVs) requires testing against rare, dangerous traffic events that are incredibly sparse in real-world datasets. This paper introduces a novel rare event sampling technique based on the "theory of bounded rationality" to model sub-optimal human driving behavior. The approach significantly speeds up simulation testing and generates realistic near-miss and crash scenarios with much lower variance and greater interpretability than standard cross-entropy methods.

What Problem Does This Paper Address?

To guarantee the safety of Autonomous Vehicles (AVs), field testing alone is statistically insufficient—an AV would need to drive billions of miles to prove its reliability. Consequently, the industry relies heavily on simulation. However, for simulations to be useful, the simulated human drivers must behave realistically.

The core problem is that real-world naturalistic driving datasets mostly contain typical, safe behaviors. "Rare events" like crashes and near-misses are, by definition, missing or extremely sparse. Existing rare event (RE) sampling methods, like Cross-Entropy (CE) importance sampling, mathematically force simulations into failure states to test the AV. However, these methods often lack interpretability; they tweak statistical distributions to cause a crash, but they don't explain why the human driver caused the crash in terms of actual driving behavior.

Methods

The authors tackle this by applying the theory of bounded rationality—a concept from behavioral economics and AI which posits that agents (human drivers) do not always make mathematically optimal decisions.

Instead of just perturbing physics or trajectories, the authors model human drivers using a set of utility functions based on safety (distance gap, time-to-collision) and progress (velocity). They introduce a rationality vector ($\Lambda$) that dictates how strictly a driver adheres to these utilities.

  1. Behavior Categorization: By varying the rationality vector, the authors categorize drivers into 8 distinct behavior profiles for a "cut-in" scenario (e.g., aggressive high-speed cut-ins vs. cautious low-speed cut-ins).
  2. Parameter Optimization: They use a Simulated Annealing (SA) heuristic to find the specific behavior category and rationality parameters that maximize the likelihood of a rare event (defined here as a near-crash where the distance gap drops below 0.01 meters).
  3. Situation Generation: They extend the model with mixing parameters to blend different behavior categories, allowing the model to generate synthetic data that mimics the multimodal nature of real-world human driving.

Datasets

  • University of Michigan SPMD (Safety Pilot Model Deployment) dataset: A massive naturalistic driving dataset collected over two years.
  • The authors extracted 74,449 vehicle cut-in events from this dataset, capturing the trajectories of both the target and subject vehicles for 5 seconds immediately following the cut-in maneuver.

Key Insights and Ideas

  • Behavioral Interpretability: Unlike standard Cross-Entropy methods that operate purely on statistical distributions, this bounded rationality method tweaks human motivations. This makes the generated edge cases highly interpretable (e.g., an engineer can see that a simulated crash happened specifically because the simulated driver prioritized speed over time-to-collision).
  • Sub-optimality is the Key to Edge Cases: Pure utility maximization (perfectly rational driving) rarely causes the critical events AVs need to be tested against. Explicitly modeling the sub-optimality of human decisions is crucial for realistic safety verification.
  • Better Proposals Mean Lower Variance: In importance sampling, the goal is to find a "proposal distribution" that frequently generates rare events but remains mathematically close to the real-world distribution. By grounding their proposal distribution in actual behavior categories, the authors achieved a drastically lower variance than CE methods, leading to much tighter, more reliable estimates of true rare event probabilities.

Key Results

  • Massive Speedups: The Bounded Rationality (BR) approach is $10^4$ times faster than Crude Monte Carlo (CMC) sampling and provides a 33% speedup over standard Cross-Entropy (CE) sampling.
  • Variance Reduction: The BR method achieved a 99% reduction in variance compared to CE. While both CE and BR successfully force rare events at high rates ($\sim3-4 \times 10^{-2}$ compared to CMC's $5.6 \times 10^{-4}$), BR's lower variance means its mathematical estimate of the true probability of these rare events is far more accurate.
  • High Fidelity to Real Data: When generating synthetic driving situations, the BR model strongly correlated with real-world naturalistic driving data. QQ plots showed a strong linear relation, with Pearson correlation coefficients of $\rho = 0.919$ for time-to-collision and $\rho = 0.98$ for distance gap.

Limitations and Future Work

While highly effective, the paper primarily demonstrates the approach on a single specific maneuver: two-lane vehicle cut-ins. The utility functions (distance gap, time-to-collision, velocity) are manually defined for this specific scenario.

Though not explicitly detailed in a dedicated "Future Work" section, the natural next step for this research is expanding the bounded rationality framework to a wider variety of complex urban situations—such as multi-agent intersection negotiations or interactions with pedestrians—where defining the utility functions and behavior categories will be significantly more complex.

Summary updated

Abstract

Performance evaluation of urban autonomous vehicles (AVs) requires a realistic model of the behavior of other road users in the environment. Learning such models from data involves collecting naturalistic data of real-world human behavior. In many cases, acquisition of this data can be prohibitively expensive or intrusive. Additionally, the available data often contain only typical behaviors and exclude behaviors that are classified as rare events. To evaluate the performance of AVs in such situations, we develop a model of traffic behavior based on the theory of bounded rationality. Based on the experiments performed on a large naturalistic driving data, we show that the developed model can be applied to estimate probability of rare events, as well as to generate new traffic situations for testing.
BibTeX
@inproceedings{sarkar2019behavior,
  author = {Atrisha Sarkar and Krzysztof Czarnecki},
  title = {{A behavior driven approach for sampling rare event situations for autonomous vehicles}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=WON2O97MJ7},
}

Loading related content...

Figure