Leveraging on Deep Reinforcement Learning for Autonomous Safe Decision-Making in Highway On-ramp Merging (Student Abstract)

Zine El Abidine Kherroubi, Samir Aknine, Rebiha BachaAAAI 2021 · DOI

Introduction

Highway on-ramp merging represents a significant challenge for autonomous vehicles (AVs), requiring high-speed coordination and cooperation with human drivers. Traditional Reinforcement Learning (RL) approaches often treat surrounding traffic as static obstacles, failing to account for the dynamic and often unpredictable reactions of human drivers. This research introduces a novel framework that combines Deep Deterministic Policy Gradient (DDPG) with an intention prediction module to facilitate safer and more efficient merging in mixed traffic environments.

Methodology

The proposed system, designated as DDPG-I, utilizes a continuous action space architecture to ensure smooth longitudinal acceleration. The core innovation lies in the integration of an Artificial Neural Network (ANN) that predicts the intentions of human drivers in the main lane.

The merging scenario layout, illustrating the DDPG control zone and the relative positions of the merging vehicle (Vm) and surrounding highway vehicles (P1, P2, F1, F2).

As shown in Figure 1, the agent monitors a specific "control zone" using Vehicle-to-Infrastructure (V2I) communication. The state input includes the positions and velocities of the merging vehicle's projection ($V_m$), two preceding vehicles ($P_1, P_2$), and two following vehicles ($F_1, F_2$). By incorporating the predicted driver intention into the state vector, the agent can proactively adjust its speed to yield or merge based on whether a human driver is likely to cooperate.

Experimental Setup and Results

The framework was evaluated using the Simulation of Urban Mobility (SUMO) environment, modeled after the I-80 highway section in Emeryville, California. The DDPG-I agent was compared against a baseline DDPG model without intention prediction.

Comparison of average undiscounted rewards over training episodes, demonstrating that the DDPG-I model (red) converges significantly faster and to a higher reward than the standard DDPG (blue).

Key Performance Metrics

  • Learning Efficiency: The inclusion of driver intention acted as a catalyst for learning. DDPG-I reached an optimal policy in approximately 450,000 steps, a 55% reduction compared to the 1,000,000 steps required by the baseline.
  • Safety and Cooperation: The DDPG-I agent achieved zero collisions across all test episodes. Furthermore, it significantly reduced the need for emergency braking by following vehicles (from 8 instances down to 3) and maintained a higher average merging speed (26 m/s vs. 23.4 m/s).
  • Speed Matching: The intention-aware model demonstrated better synchronization with traffic flow, matching the lead vehicle's speed with an average difference of only 7.8 m/s, compared to 10.5 m/s for the baseline.

Conclusion

By "reading the room" and anticipating human behavior, the DDPG-I framework transforms the merging process from a reactive obstacle-avoidance task into a cooperative maneuver. While the current model assumes full observability through infrastructure support, future work will focus on adapting the system for decentralized environments where AVs must rely on onboard sensors and handle varying traffic densities. This research underscores the necessity of human-centric AI for the successful integration of autonomous vehicles into today's roads.

Summary updated

BibTeX
@inproceedings{kherroubi2021leveraging,
  author = {Zine El Abidine Kherroubi and Samir Aknine and Rebiha Bacha},
  title = {{Leveraging on Deep Reinforcement Learning for Autonomous Safe Decision-Making in Highway On-ramp Merging (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17904},
}
Figure