Abstract
Applying reinforcement learning to autonomous driving has garnered widespread attention. However, classical reinforcement learning methods optimize policies by maximizing expected rewards but lack sufficient safety considerations, often putting agents in hazardous situations. This paper proposes a risk-aware reinforcement learning approach for autonomous driving to improve the safety performance when crossing the intersection. Safe critics are constructed to evaluate driving risk and work in conjunction with the reward critic to update the actor. Based on this, a Lagrangian relaxation method and cyclic gradient iteration are combined to project actions into a feasible safe region. Furthermore, a Multi-hop and Multi-layer perception (MLP) mixed Attention Mechanism (MMAM) is incorporated into the actor-critic network, enabling the policy to adapt to dynamic traffic and overcome permutation sensitivity challenges. This allows the policy to focus more effectively on surrounding potential risks while enhancing the identification of passing opportunities. Simulation tests are conducted on different tasks at unsignalized intersections. The results show that the proposed approach effectively reduces collision rates and improves crossing efficiency in comparison to baseline algorithms. Additionally, our ablation experiments demonstrate the benefits of incorporating risk-awareness and MMAM into RL.
Introduction
Navigating unsignalized intersections represents a significant challenge for autonomous vehicles (AVs), requiring complex social negotiation and precise timing. While Reinforcement Learning (RL) has shown potential in mastering these maneuvers, standard RL agents often suffer from "safety blindness," learning through repeated collisions and failing to generalize safety constraints in dynamic environments. Furthermore, traditional neural architectures like Multi-Layer Perceptrons (MLPs) struggle with the varying number of surrounding vehicles (dimension sensitivity) and the arbitrary order of vehicle detection (permutation sensitivity).
To address these issues, we introduce ARSAC (Attention-embedded and Risk-aware Soft Actor-Critic). This framework moves beyond simple negative rewards for collisions by treating safety as a hard constraint, mathematically defining a "feasible safe region" ($S_{FR}$) for vehicle actions.
Methodology
The ARSAC framework integrates three core components to ensure safe and efficient navigation:
1. Safe Critics and Lagrangian Relaxation
The system utilizes dual critics: a reward critic to optimize for task completion and a Safe Critic to evaluate the risk of state-action pairs based on predicted trajectories. By applying Lagrangian relaxation, the agent identifies an approximate safe action that balances reward maximization with safety constraints.
2. Safety Iterative Correction
When the actor proposes an action deemed risky by the Safe Critic, a Safety Iterative Correction module employs cyclic gradient descent. This process "nudges" the initial action ($a_{init}$) into the feasible safe region ($S_{FR}$) before execution, ensuring the vehicle remains within safe operational bounds.
3. Multi-hop and MLP-mixed Attention Mechanism (MMAM)
To handle dynamic traffic, a 2-hop attention structure is integrated into the actor-critic network. The first hop (Self-Attention) establishes spatial relationships between surrounding vehicles, while the second hop (Ego-Attention) focuses the ego vehicle's "attention" on the most critical threats. This mechanism ensures the model is permutation-invariant and can effectively ignore non-conflicting traffic noise.
Experimental Setup
Evaluations were conducted in the Highway-Env simulation, featuring a bidirectional four-lane unsignalized intersection. The agent was tested on three maneuvers: Left-Turn (LT), Go-Straight (GS), and Right-Turn (RT). Surrounding vehicles were controlled by an improved Intelligent Driver Model (IDM) to simulate realistic human behavior, and the ego vehicle utilized a discrete dynamic bicycle model for stability.
Results and Discussion
ARSAC was compared against baselines including PPO-RS, SAC-Lag, and CPO. In the challenging unprotected Left-Turn task, ARSAC achieved a collision rate (CR) of 2.8%, a significant improvement over SAC-Lag (17.4%) and CPO (38.4%). The success rate (SR) reached 95.8%, compared to the 61-80% range of baselines.
Crucially, ARSAC avoided the "frozen robot" problem. The Frozen Rate (FR)—measuring instances where the vehicle remains stationary for excessive periods—remained low at 1.4% for left turns. Ablation studies confirmed that both the MMAM (attention) and the risk-aware safe critic are essential for achieving these performance gains.
Conclusion
The proposed risk-aware RL approach significantly enhances safety at unsignalized intersections without sacrificing efficiency. By combining safety projections with advanced attention mechanisms, ARSAC provides a robust framework for autonomous navigation. Future work will focus on incorporating vulnerable road users (cyclists, pedestrians) and bridging the sim-to-real gap through training on real-world datasets.
Summary updated