A Comparative Study of Deep Reinforcement Learning-based Transferable Energy Management Strategies for Hybrid Electric Vehicles

Jingyi Xu, Zirui Li, Li Gao, Junyi Ma, Qi Liu, Yanan Zhao

TL;DR

This paper investigates how different noise-injection strategies in Deep Reinforcement Learning (DRL) affect the efficiency of transferring energy management strategies (EMS) to new driving conditions in hybrid electric vehicles (HEVs). They found that injecting noise directly into the DRL agent's parameters (parameter space noise) leads to more stable and faster learning when transferring knowledge to new driving cycles, outperforming action space noise or combined noise strategies.

What Problem Does This Paper Address?

Hybrid Electric Vehicles (HEVs) rely on sophisticated Energy Management Strategies (EMS) to optimize power distribution between their electric and combustion sources, thereby improving fuel efficiency and reducing emissions. Deep Reinforcement Learning (DRL) has emerged as a powerful tool for developing these EMS, as it can learn optimal control policies through interaction with the environment. However, a significant drawback of DRL is its lengthy training time. Furthermore, when the driving conditions (e.g., driving cycles) change, the DRL model typically needs to be retrained from scratch, which is computationally expensive and time-consuming. This paper addresses this challenge by exploring how transfer learning, combined with different DRL exploration techniques, can accelerate the adaptation of EMS to new driving cycles. Specifically, they aim to identify the most effective exploration strategy for DRL-based transferable EMS.

Methods

The core of this work lies in comparing different exploration methods within the Deep Deterministic Policy Gradient (DDPG) algorithm, a popular actor-critic DRL framework. DDPG uses an actor network to select actions and a critic network to evaluate those actions. To encourage exploration and prevent the agent from getting stuck in local optima, noise is typically added. This paper focuses on two primary types of noise injection:

  • Action Space Noise: Noise is added directly to the actions selected by the actor network. This can be a simple Gaussian noise or a more complex Ornstein-Uhlenbeck (OU) process for temporally correlated noise.
  • Parameter Space Noise: Noise is added directly to the parameters of the actor network. This aims to create more consistent exploration by perturbing the underlying policy directly.

The researchers also investigate a combination of both action and parameter space noise. They train DDPG agents with these different noise strategies on a set of "source" driving cycles. The learned parameters from these source-trained agents are then used to initialize DDPG agents that are subsequently trained on a different "target" driving cycle. The effectiveness of transfer learning is evaluated based on how quickly and effectively the target agent learns an optimal policy.

Datasets

The paper utilizes standard driving cycles for training and evaluation:

  • Source Domain: Urban Dynamometer Driving Schedule (UDDS) and FTP75.
  • Target Domain: New European Driving Cycle (NEDC).

The authors note that the target driving cycle (NEDC) is different from, but similar to, the source driving cycles, which is a common requirement for successful transfer learning.

Key Insights and Ideas

  • Parameter Space Noise is Key for Transfer: The most significant finding is that injecting noise directly into the DRL agent's parameters (parameter space noise) leads to the most stable and fastest convergence when transferring learned EMS policies to new driving cycles. This suggests that perturbing the policy's underlying structure is more beneficial for adaptation than simply adding noise to the actions.
  • Action Space Noise Offers Stability in Source Domain: While parameter space noise excels in transfer, action space noise (particularly Gaussian) demonstrated good stability during training within the source domain.
  • Combined Noise Can Be Detrimental: Counter-intuitively, combining action space and parameter space noise often performed poorly, especially in the transfer learning phase. This might be due to excessive exploration that hinders efficient learning or creates conflicting exploration signals.
  • Importance of Source Domain Training: Training the DRL agent on a diverse set of source driving cycles improves its generalization ability, which is crucial for effective knowledge transfer to new, unseen driving conditions.

Key Results

The paper presents quantitative results in Table IV and qualitative observations in Figure 5.

  • Jumpstart Performance (JP): When transferring knowledge, networks initialized with parameter space noise generally showed better initial performance (higher mean reward in the first 50 episodes) compared to those trained from scratch or initialized with action space noise.
  • Asymptotic Performance (AP): After convergence, the mean reward in the convergence interval was comparable across most methods, suggesting that while parameter space noise accelerates learning, it doesn't necessarily lead to a significantly higher ultimate performance ceiling compared to other well-performing methods.
  • Time to Threshold (TT): Parameter space noise consistently resulted in a lower iteration number for convergence in the target domain, indicating faster learning and thus improved transfer efficiency. For instance, using Gaussian noise in the parameter space for initialization and OU noise for exploration in the target domain achieved convergence in 22 iterations, significantly faster than training from scratch (36 iterations).
  • Stability: Visualizations in Figure 5 clearly show that parameter space noise-initialized target networks exhibit less reward fluctuation during training, indicating greater stability.

Limitations and Future Work

  • Limited Exploration Modalities: The study primarily focuses on adding noise for action selection and parameter perturbation. The authors suggest exploring other exploration modalities beyond simple noise injection.
  • Comparison with Other DRL Algorithms: The current study uses DDPG. Future work could involve comparing these exploration strategies with more advanced DRL algorithms like Twin Delayed DDPG (TD3) and Soft Actor-Critic (SAC).
  • Robustness of Transfer: While parameter space noise showed promise, further investigation into the robustness of transfer learning across even more diverse driving cycles or vehicle types could be beneficial.
  • Code Availability: The authors have made their code publicly available, which is a positive step for reproducibility and further research.

Summary updated

Abstract

The deep reinforcement learning-based energy management strategies (EMS) have become a promising solution for hybrid electric vehicles (HEVs). When driving cycles are changed, the neural network will be retrained, which is a time-consuming and laborious task. A more efficient way of choosing EMS is to combine deep reinforcement learning (DRL) with transfer learning, which can transfer knowledge of one domain to the other new domain, making the network of the new domain reach convergence values quickly. Different exploration methods of DRL, including adding action space noise and parameter space noise, are compared against each other in the transfer learning process in this work. Results indicate that the network added parameter space noise is more stable and faster convergent than the others. In conclusion, the best exploration method for transferable EMS is to add noise in the parameter space, while the combination of action space noise and parameter space noise generally performs poorly. Our code is available at https://github.com/BIT-XJY/RL-based-Transferable-EMS.git.
BibTeX
@inproceedings{xu2022comparative,
  author = {Jingyi Xu and Zirui Li and Li Gao and Junyi Ma and Qi Liu and Yanan Zhao},
  title = {{A Comparative Study of Deep Reinforcement Learning-based Transferable Energy Management Strategies for Hybrid Electric Vehicles}},
  booktitle = {2022},
  year = {2022},
  doi = {10.1109/iv51971.2022.9827042},
}
Figure