Co-Optimization of On-Ramp Merging and Plug-In Hybrid Electric Vehicle Power Split Using Deep Reinforcement Learning

Yuan Lin, John McPhee, Nasser L. Azad

Abstract

Current research on Deep Reinforcement Learning (DRL) for automated on-ramp merging often neglects the complexities of vehicle powertrain dynamics. This work addresses this gap by considering automated on-ramp merging for a power-split Plug-In Hybrid Electric Vehicle (PHEV), specifically the 2015 Toyota Prius Plug-In. We co-optimize the on-ramp merging control and PHEV energy management such that the DRL policy directly determines the power split between the engine and the electric motor. Testing results demonstrate that DRL-based co-optimization leads to collision-free merging and improved energy efficiency. While co-optimization results in more economic performance, it introduces slightly higher jerk compared to sequential approaches, which may suffer from collisions due to neglecting powertrain power limits.

Introduction

Automated highway merging is a critical challenge for autonomous vehicles. Most existing research treats vehicles as simple point-masses, ignoring the physical constraints of the powertrain. In reality, a vehicle's acceleration is bounded by its engine and motor capabilities. For Plug-In Hybrid Electric Vehicles (PHEVs), the problem is further complicated by the need to manage energy between two power sources. If a merging controller requests acceleration beyond what the powertrain can deliver, safety is compromised. This study proposes a co-optimization framework using Deep Reinforcement Learning to bridge the gap between high-level traffic maneuvering and low-level mechanical constraints.

Methodology

The core of this research is the shift from sequential control to co-optimization. In traditional sequential approaches, an upper-level controller determines the required acceleration, and a lower-level controller manages the power split. In our co-optimization approach, a single DRL agent processes both traffic data and internal vehicle states (like battery charge) to output the power split directly.

Comparison between co-optimization and sequential control architectures for PHEV on-ramp merging.

We utilized the Soft Actor-Critic (SAC) algorithm, a DRL method suited for continuous action spaces. The vehicle model is based on the 2015 Toyota Prius Plug-In, incorporating its planetary gear system and battery dynamics. The DRL agent's reward function balances safety (penalizing collisions), comfort (penalizing jerk), and efficiency (penalizing energy cost).

Experimental Setup

The system was evaluated using the SUMO (Simulation of Urban Mobility) simulator. The environment consists of a highway with a single on-ramp. Traffic on the main road is generated using the Intelligent Driver Model (IDM) to simulate realistic vehicle interactions.

Simulation environment showing the control zone and sensing space for the merging vehicle.

The agent was trained over 1 million time steps, covering approximately 13,000 unique merging scenarios with varying traffic densities and initial vehicle states. The "Sensing Space" extends 200m to allow the agent to perceive surrounding traffic before entering the "Control Zone."

Results and Discussion

The co-optimized DRL agent achieved a 0% collision rate, significantly outperforming the acceleration-based sequential approach, which had a 0.08% collision rate due to "power saturation"—situations where the requested acceleration exceeded the vehicle's physical limits.

In terms of efficiency, the co-optimized approach reduced episodic energy costs by 8% compared to sequential baselines. The agent learned to maximize regenerative braking, treating electricity as a "negative cost." However, this optimization came at a cost to ride comfort; the co-optimized maneuvers were 11% jerkier than the baseline, highlighting a trade-off between mathematical efficiency and human-centric comfort.

Furthermore, the trained DRL policy is computationally lightweight. The neural network's small size allows it to run in real-time on standard automotive Electronic Control Units (ECUs), offering a faster alternative to traditional Model Predictive Control (MPC) methods.

Conclusion

This work demonstrates that co-optimizing traffic maneuvers with powertrain management using DRL enhances both safety and energy efficiency for PHEVs. By making the "merging brain" aware of mechanical limits, we eliminate collisions caused by power saturation. Future research will focus on reducing jerkiness and testing these policies in more complex multi-lane environments and real-world conditions.

Summary updated

Abstract

Current research on Deep Reinforcement Learning (DRL) for automated on-ramp merging neglects vehicle powertrain and dynamics. This work considers automated on-ramp merging for a power-split Plug-In Hybrid Electric Vehicle (PHEV), the 2015 Toyota Prius Plug-In, using DRL. The on-ramp merging control and the PHEV energy management are co-optimized such that the DRL policy directly outputs the power split between the engine and the electric motor. The testing results show that DRL can be successfully used for co-optimization, leading to collision-free on-ramp merging. When compared with sequential approaches wherein the upper-level on-ramp merging control and the lower-level PHEV energy management are performed independently and in sequence, we found that co-optimization results in economic but jerky on-ramp merging while sequential approaches may result in collisions due to neglecting powertrain power limit constraints in designing the upper-level on-ramp merging controller.
BibTeX
@article{lin2022co,
  author = {Yuan Lin and John McPhee and Nasser L. Azad},
  title = {{Co-Optimization of On-Ramp Merging and Plug-In Hybrid Electric Vehicle Power Split Using Deep Reinforcement Learning}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2022},
  url = {https://openreview.net/forum?id=aZLSq6w5Pz},
}
Figure