Abstract
Current research on Deep Reinforcement Learning (DRL) for automated on-ramp merging often neglects the complexities of vehicle powertrain dynamics. This work addresses this gap by considering automated on-ramp merging for a power-split Plug-In Hybrid Electric Vehicle (PHEV), specifically the 2015 Toyota Prius Plug-In. We co-optimize the on-ramp merging control and PHEV energy management such that the DRL policy directly determines the power split between the engine and the electric motor. Testing results demonstrate that DRL-based co-optimization leads to collision-free merging and improved energy efficiency. While co-optimization results in more economic performance, it introduces slightly higher jerk compared to sequential approaches, which may suffer from collisions due to neglecting powertrain power limits.
Introduction
Automated highway merging is a critical challenge for autonomous vehicles. Most existing research treats vehicles as simple point-masses, ignoring the physical constraints of the powertrain. In reality, a vehicle's acceleration is bounded by its engine and motor capabilities. For Plug-In Hybrid Electric Vehicles (PHEVs), the problem is further complicated by the need to manage energy between two power sources. If a merging controller requests acceleration beyond what the powertrain can deliver, safety is compromised. This study proposes a co-optimization framework using Deep Reinforcement Learning to bridge the gap between high-level traffic maneuvering and low-level mechanical constraints.
Methodology
The core of this research is the shift from sequential control to co-optimization. In traditional sequential approaches, an upper-level controller determines the required acceleration, and a lower-level controller manages the power split. In our co-optimization approach, a single DRL agent processes both traffic data and internal vehicle states (like battery charge) to output the power split directly.
We utilized the Soft Actor-Critic (SAC) algorithm, a DRL method suited for continuous action spaces. The vehicle model is based on the 2015 Toyota Prius Plug-In, incorporating its planetary gear system and battery dynamics. The DRL agent's reward function balances safety (penalizing collisions), comfort (penalizing jerk), and efficiency (penalizing energy cost).
Experimental Setup
The system was evaluated using the SUMO (Simulation of Urban Mobility) simulator. The environment consists of a highway with a single on-ramp. Traffic on the main road is generated using the Intelligent Driver Model (IDM) to simulate realistic vehicle interactions.
The agent was trained over 1 million time steps, covering approximately 13,000 unique merging scenarios with varying traffic densities and initial vehicle states. The "Sensing Space" extends 200m to allow the agent to perceive surrounding traffic before entering the "Control Zone."
Results and Discussion
The co-optimized DRL agent achieved a 0% collision rate, significantly outperforming the acceleration-based sequential approach, which had a 0.08% collision rate due to "power saturation"—situations where the requested acceleration exceeded the vehicle's physical limits.
In terms of efficiency, the co-optimized approach reduced episodic energy costs by 8% compared to sequential baselines. The agent learned to maximize regenerative braking, treating electricity as a "negative cost." However, this optimization came at a cost to ride comfort; the co-optimized maneuvers were 11% jerkier than the baseline, highlighting a trade-off between mathematical efficiency and human-centric comfort.
Furthermore, the trained DRL policy is computationally lightweight. The neural network's small size allows it to run in real-time on standard automotive Electronic Control Units (ECUs), offering a faster alternative to traditional Model Predictive Control (MPC) methods.
Conclusion
This work demonstrates that co-optimizing traffic maneuvers with powertrain management using DRL enhances both safety and energy efficiency for PHEVs. By making the "merging brain" aware of mechanical limits, we eliminate collisions caused by power saturation. Future research will focus on reducing jerkiness and testing these policies in more complex multi-lane environments and real-world conditions.
Summary updated