TL;DR
This paper introduces a novel hierarchical deep reinforcement learning (DRL) approach for optimizing energy management in hybrid electric vehicles (HEVs). By combining Soft Actor-Critic (SAC) for continuous engine torque control and Deep Q-Network (DQN) for discrete gear shifting, the proposed system achieves near-optimal fuel efficiency comparable to traditional methods while significantly improving computational efficiency for real-time deployment.
What Problem Does This Paper Address?
Hybrid electric vehicles (HEVs) offer a promising solution for improving fuel economy and reducing emissions. However, their efficient operation hinges on sophisticated energy management strategies (EMS) that can intelligently coordinate multiple power sources (engine and electric motor) and control systems. A major challenge in designing these EMSs, especially with deep reinforcement learning (DRL), is handling the complex interplay of both continuous (e.g., engine torque) and discrete (e.g., gear selection) control variables simultaneously. Traditional DRL methods often struggle with this "hybrid action space," leading to suboptimal performance, reduced efficiency, and difficulties in real-time implementation. This paper aims to bridge this gap by developing a hierarchical DRL framework specifically designed to tackle these hybrid control challenges in HEVs.
Methods
The core of this paper is a Hierarchical Reinforcement Learning (HRL) framework that separates the control tasks into two layers:
- Upper Layer (Continuous Control): This layer uses the Soft Actor-Critic (SAC) algorithm to manage the continuous control of the engine's output torque. SAC is chosen for its ability to handle continuous action spaces and its incorporation of entropy regularization, which encourages exploration and helps avoid premature convergence to suboptimal solutions.
- Lower Layer (Discrete Control): This layer employs the Deep Q-Network (DQN) algorithm to optimize discrete gear selection for the transmission. DQN is a well-established DRL algorithm for discrete action spaces, known for its ability to approximate state-action value functions using neural networks.
This hierarchical structure allows each algorithm to focus on its respective control domain, with a shared reward signal guiding both the upper and lower levels towards the overall objective of maximizing fuel economy and maintaining battery state-of-charge (SOC). The system is trained offline and then tested online.
Datasets
- Training: The China-World Transient Vehicle Cycle (C-WTVC) is used for offline training. This cycle is 20.5 km long and lasts for 1802 seconds, providing representative power demand characteristics for learning.
- Testing/Validation:
- The Orange County Transit Bus Cycle (US-OCTA) is used for online testing. This cycle simulates urban bus operations with frequent acceleration and deceleration, making it suitable for evaluating gear-shifting strategies.
- Real-world bus driving cycles from the U.S. National Renewable Energy Laboratory's Drive Cycle Analysis Tool are used for adaptability validation, covering prolonged and complex driving conditions.
Key Insights and Ideas
- Hybrid Action Space Management: The paper demonstrates that a hierarchical approach, where specialized DRL algorithms handle discrete and continuous actions separately, is highly effective for complex HEV control. This avoids the pitfalls of trying to force hybrid actions into a single algorithm or using cumbersome transformations.
- Synergy of SAC and DQN: The combination of SAC for continuous torque control and DQN for discrete gear shifting creates a powerful synergy. SAC's exploration capabilities and DQN's discrete optimization work together to achieve a more coordinated and efficient powertrain operation.
- Shared Reward for Coordinated Control: Using a single, unified reward signal for both the upper and lower-level controllers ensures that their independent actions are aligned towards the common goal of fuel economy and battery management.
- Adaptability to Real-World Conditions: The proposed strategy shows promising adaptability, maintaining stable performance and effective energy management even under prolonged and varied driving scenarios, which is crucial for practical deployment.
Key Results
- Fuel Economy: The proposed SAC/DQN-based EMS achieves fuel economy performance comparable to Dynamic Programming (DP), with only a 3.06% fuel consumption gap. This is a significant achievement, as DP is known for finding near-global optimal solutions but is computationally prohibitive for real-time use.
- Computational Efficiency: The HRL approach significantly improves computational efficiency, achieving a 66.49% improvement compared to the DP/DP-based EMS. This makes real-time deployment much more feasible.
- Gear Shifting: The strategy demonstrates a balanced approach to gear shifting, avoiding the excessive shifts of DP (leading to comfort issues) and the potentially less adaptive nature of rule-based systems. It shows smooth and adaptive gear transitions that closely follow vehicle speed variations.
- SOC Stability: The battery SOC trajectories remain stable and close to the target reference value, indicating effective battery management and preventing deep discharge or overcharging.
Limitations and Future Work
- Simulation Environment: While the study validates the strategy's effectiveness and real-time potential in a simulation environment, further validation on Hardware-in-the-Loop (HIL) platforms and real vehicles is necessary to assess robustness and control stability under dynamic, non-ideal conditions.
- Richer Inputs for Adaptability: To further enhance adaptability to real-world driving, future work could incorporate richer, high-dimensional inputs such as road gradients, traffic information, and driver behavior features.
- Online Learning and Transfer Learning: Integrating online learning mechanisms and transfer learning capabilities could support adaptive control and continuous optimization of hybrid powertrains across different vehicle types and operational contexts, broadening the applicability of the approach.
Summary updated