Driving-Policy Adaptive Safeguard for Autonomous Vehicles Using Reinforcement Learning

Zhong Cao, Shaobing Xu, Songan Zhang, Huei Peng, Diange YangarXiv 2020 · arXiv

Abstract

Autonomous vehicles (AVs) typically rely on a two-layer control stack: a high-level driving policy for navigation and a low-level safeguard for emergency maneuvers. However, traditional safeguards are often policy-agnostic, leading to frequent unnecessary interventions or delayed responses. This paper introduces the Driving-Policy Adaptive Safeguard (DPAS), a reinforcement learning-based framework that learns to intervene only when the primary driving policy is predicted to fail. By integrating Monte-Carlo Tree Search (MCTS) with the vehicle's active driving policy and employing particle filters for surrounding vehicle intent estimation, DPAS achieves a significant reduction in both collision rates and "phantom" interventions. In high-fidelity highway simulations, DPAS outperformed industry benchmarks, reducing collisions by up to 35% and interventions by over 50%.

Introduction

The safety of autonomous vehicles is paramount, yet current safety mechanisms often suffer from a lack of context regarding the vehicle's intended maneuvers. Most AVs utilize a high-level driving policy for lane changes and car-following, backed by a low-level safeguard like Automatic Emergency Braking (AEB). A critical issue arises because these safeguards are generally policy-agnostic; they may trigger a hard brake based on perceived danger, even if the primary driving policy is managing the situation safely. This results in "phantom" interventions that disrupt traffic and annoy passengers, or conversely, a "safety lag" where the safeguard fails to act early enough because it doesn't recognize the specific limitations of the active driving policy.

Methodology

The DPAS framework addresses these gaps by acting as an "intelligent critic" of the primary driving policy. The core of the system is a Monte-Carlo Tree Search (MCTS) module that uses the AV’s current driving policy for its "rollout" simulations. This allows the safeguard to predict whether the current plan will lead to a collision.

The DPAS Agent architecture, illustrating the interaction between the RL Policy, the primary Driving Policy, and the Activation Function that determines the final action.

To handle the inherent uncertainty of human behavior, DPAS incorporates a Particle Filter for intent estimation. This filter estimates internal parameters of surrounding vehicles, such as aggressiveness or politeness, allowing the AV to anticipate risky maneuvers like sudden cut-ins. The safeguard is triggered by an activation function that compares the expected safety ($Q$-value) of the current driving policy against the safest alternative action identified by the MCTS. Unlike traditional systems that only control braking, DPAS can manipulate both braking and steering to navigate out of dangerous scenarios.

Experimental Setup

The researchers evaluated DPAS using a high-fidelity stochastic highway simulation. To ensure robust testing, the team conducted 90,000 simulation rounds in a three-lane environment. They employed importance sampling to "twist" traffic behavior, exposing the AV to high-risk, near-crash scenarios more frequently than they occur in standard driving. The simulated agents were calibrated using real-world naturalistic driving data to ensure their aggressive behaviors mirrored actual human driving risks.

Results and Discussion

DPAS was benchmarked against industry standards, including Responsibility Sensitive Safety (RSS) and a Reachable Set method.

Comparison of collision rates over simulation time between the standard RSS benchmark and the proposed DPAS framework.

The results indicate that DPAS significantly improves both safety and driving smoothness. Across various driving policies, DPAS reduced collision rates by 11% to 35% compared to the benchmarks. More importantly, the intervention rate—the frequency with which the safeguard overrides the primary policy—dropped by 37% to 52%. This reduction in "phantom" braking events led to a 5% increase in average vehicle speed, demonstrating that a smarter safeguard can improve overall traffic efficiency. When mapped to real-world statistics, DPAS showed a decrease of approximately 3.2 collisions per million kilometers driven compared to RSS.

Conclusion

The Driving-Policy Adaptive Safeguard represents a shift from reactive, policy-agnostic safety layers to proactive, context-aware systems. By understanding the "intent" of the primary driver, DPAS minimizes unnecessary disruptions while providing superior protection in critical moments. Future work will focus on extending this framework to complex urban environments involving intersections and pedestrians, as well as optimizing the computational efficiency of MCTS for real-time deployment on production hardware.

Summary updated

Abstract

Safeguard functions such as those provided by advanced emergency braking (AEB) can provide another layer of safety for autonomous vehicles (AV). A smart safeguard function should adapt the activation conditions to the driving policy, to avoid unnecessary interventions as well as improve vehicle safety. This paper proposes a driving-policy adaptive safeguard (DPAS) design, including a collision avoidance strategy and an activation function. The collision avoidance strategy is designed in a reinforcement learning framework, obtained by Monte-Carlo Tree Search (MCTS). It can learn from past collisions and manipulate both braking and steering in stochastic traffics. The driving-policy adaptive activation function should dynamically assess current driving policy risk and kick in when an urgent threat is detected. To generate this activation function, MCTS' exploration and rollout modules are designed to fully evaluate the AV's current driving policy, and then explore other safer actions. In this study, the DPAS is validated with two typical highway-driving policies. The results are obtained through and 90,000 times in the stochastic and aggressive simulated traffic. The results are calibrated by naturalistic driving data and show that the proposed safeguard reduces the collision rate significantly without introducing more interventions, compared with the state-based benchmark safeguards. In summary, the proposed safeguard leverages the learning-based method in stochastic and emergent scenarios and imposes minimal influence on the driving policy.
BibTeX
@misc{cao2020driving,
  author = {Zhong Cao and Shaobing Xu and Songan Zhang and Huei Peng and Diange Yang},
  title = {{Driving-Policy Adaptive Safeguard for Autonomous Vehicles Using Reinforcement Learning}},
  year = {2020},
  eprint = {2012.01010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.01010},
}
Figure