Deep Reinforcement Learning-Based Battery Conditioning Hierarchical V2G Coordination for Multi-Stakeholder Benefits

Yubao Zhang, Xin Chen, Yi Gu, Zhicheng Li, Wu KaiarXiv 2023 · DOI · arXiv

Abstract

This paper proposes a hierarchical coordination strategy for Vehicle-to-Grid (V2G) systems that leverages Deep Reinforcement Learning (DRL) and a Proof of Stake (PoS) algorithm to balance the competing interests of the power grid, aggregators, and electric vehicle (EV) owners. By integrating detailed battery health metrics—State of Health (SOH), State of Power (SOP), and State of Charge (SOC)—into the decision-making process, the system stabilizes the grid and reduces costs without sacrificing the long-term lifespan of vehicle batteries.

Introduction

Vehicle-to-Grid (V2G) technology offers a promising solution for grid stability by using electric vehicles as mobile energy storage units. However, practical implementation faces a multi-stakeholder challenge: the power grid requires stable loads, aggregators seek to minimize costs, and EV owners demand vehicle readiness while fearing battery degradation. Existing strategies often overlook the non-linear physics of battery health or fail to manage large fleets fairly. This study introduces the Multi-Stakeholder Hierarchical V2G Coordination (MHVCS) framework to address these conflicting objectives.

Methodology

The MHVCS framework operates on two distinct levels to manage complexity and ensure physical safety:

  1. Upper Level (Aggregator Coordination): A Proximal Policy Optimization (PPO) agent manages the total energy flow between the aggregator and the grid. The agent is trained with a multi-objective reward function that penalizes grid load variance and high operational costs while rewarding renewable energy utilization and battery health preservation.
  2. Lower Level (Individual Allocation): Inspired by blockchain consensus, a Proof of Stake (PoS) algorithm distributes the aggregator's total power requirement among individual EVs. Here, "stake" is defined by a vehicle's available battery energy, ensuring a fair distribution where vehicles with higher energy "collateral" contribute more to the V2G load.
  3. Battery Conditioning (SOX Model): The system incorporates a detailed model of State of Charge (SOC), State of Power (SOP), and State of Health (SOH). This series-parallel battery array model ensures that charging commands never violate physical safety limits or accelerate degradation unnecessarily.
The hierarchical V2G coordination framework, illustrating the interaction between the PPO-based aggregator level and the individual EV units.

Experimental Setup

The framework was evaluated in a simulated microgrid environment featuring 509 EVs with stochastic arrival and departure times. The grid data included 24-hour profiles for baseload, wind, and solar power. The MHVCS model was compared against four baselines: Uncontrolled Charging (BL1), Optimal Charging (BL2), Minimum Grid Fluctuation (BL3), and Minimum Cost (BL4).

Key Results

Grid Stability and Load Management

The MHVCS strategy demonstrated superior performance in flattening the load profile. It achieved a Load Variance (LV) of 71,312.8, a significant improvement over the 508,479.1 recorded for uncontrolled charging (BL1). As shown in the load profile comparison, the MHVCS strategy successfully kept the total load within the transformer constraints during peak hours.

Comparison of 24-hour load profiles across different strategies, highlighting the load-flattening and constraint-adherence capabilities of the MHVCS framework.

Battery Longevity and Economic Efficiency

After a year of simulated operation, the MHVCS maintained a State of Health (SOH) of 97.13%. While slightly lower than pure charging strategies, it significantly outperformed the cost-minimization strategy (BL4, 96.76%), which tends to overstress batteries for profit. Economically, the system achieved a balanced total cost of $3,127.8, proving more sustainable than standard charging while providing essential grid services.

Conclusion

By combining DRL at the aggregator level with a deterministic PoS algorithm at the vehicle level, the MHVCS framework achieves high performance with low computational overhead. The inclusion of SOP and SOH constraints ensures that V2G participation does not come at the cost of battery safety or longevity, providing a viable path forward for multi-stakeholder V2G integration.

Summary updated

Abstract

With the growing prevalence of electric vehicles (EVs) and advancements in EV electronics, vehicle-to-grid (V2G) techniques and large-scale scheduling strategies have emerged to promote renewable energy utilization and power grid stability. This study proposes a multi-stakeholder hierarchical V2G coordination based on deep reinforcement learning (DRL) and the Proof of Stake algorithm. Furthermore, the multi-stakeholders include the power grid, EV aggregators (EVAs), and users, and the proposed strategy can achieve multi-stakeholder benefits. On the grid side, load fluctuations and renewable energy consumption are considered, while on the EVA side, energy constraints and charging costs are considered. The three critical battery conditioning parameters of battery SOX are considered on the user side, including state of charge, state of power, and state of health. Compared with four typical baselines, the multi-stakeholder hierarchical coordination strategy can enhance renewable energy consumption, mitigate load fluctuations, meet the energy demands of EVA, and reduce charging costs and battery degradation under realistic operating conditions.
BibTeX
@misc{zhang2023deep,
  author = {Yubao Zhang and Xin Chen and Yi Gu and Zhicheng Li and Wu Kai},
  title = {{Deep Reinforcement Learning-Based Battery Conditioning Hierarchical V2G Coordination for Multi-Stakeholder Benefits}},
  year = {2023},
  eprint = {2308.00218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.00218},
}
Figure