Decentralized Multi-agent Reinforcement Learning based State-of-Charge Balancing Strategy for Distributed Energy Storage System

Zheng Xiong, Biao Luo, Bing-Chuan Wang, Xiaodong Xu, Xiaodong Liu, Tingwen Huang

Abstract

This paper introduces Dec-MARL, a fully decentralized reinforcement learning framework designed to balance the State-of-Charge (SoC) across distributed battery systems in a microgrid. By combining multi-agent deep reinforcement learning with a novel "counterfactual demand balance" algorithm, the system ensures that batteries of varying capacities drain and charge at rates that keep them synchronized, preventing system overloads without requiring a central controller.

Introduction

In modern microgrids, Distributed Energy Storage Systems (DESS) are vital for smoothing out the intermittent nature of renewable energy. However, these batteries often have different capacities and initial charge levels. Poor management leads to rapid depletion of some units, forcing others to compensate and causing potential system failures. Traditional centralized control methods suffer from single points of failure and poor scalability. While Multi-Agent Reinforcement Learning (MARL) offers a solution, most existing versions rely on Centralized Training, Decentralized Execution (CTDE). This paper proposes a Decentralized Training, Decentralized Execution (DTDE) approach, where agents learn and act entirely based on local communication.

Methodology

The proposed Dec-MARL framework relies on three primary pillars:

  1. Observation Expansion via Consensus: Agents use a first-order average consensus algorithm to estimate global average SoC and total power demand by communicating with immediate neighbors.
  2. Counterfactual Demand Balance (CDB) Algorithm: This core innovation manages the global constraint that total power output must match total demand. Unlike standard factual clipping, the counterfactual approach allows for better exploration of the action space, accelerating learning.
  3. Deep Deterministic Policy Gradient (DDPG): Each battery unit operates as an independent RL agent using a DDPG architecture to decide its power output based on expanded observations.
The Counterfactual Demand Balance (CDB) algorithm within the Dec-MARL framework.

Experimental Setup

The study utilizes a custom-built simulation environment of a microgrid with 5 Energy Storage Units (ESUs) with capacities ranging from 700kWh to 1800kWh. Training used a sinusoidal power demand to simulate daily fluctuations, while performance testing used linear ramp-up and constant demand profiles to evaluate adaptation to changing conditions.

Key Results and Discussion

The Dec-MARL system demonstrated successful convergence after approximately 6,000 training episodes. Key findings include:

  • Robust Decentralization: Using the Metropolis-Hastings weight matrix for communication, the system avoids a central controller, making it resilient to failures or the addition of new units.
  • Effective Balancing: Dec-MARL significantly reduced SoC variance compared to proportional allocation baselines, synchronizing all five batteries to a similar SoC (roughly 0.8).
  • Constraint Satisfaction: The CDB algorithm effectively managed global power demand constraints locally, proving that independent agents can satisfy global requirements through counterfactual exploration.
Communication topology and training performance metrics including Reward and SoC Variance.

Conclusion

The Dec-MARL framework provides a scalable and robust solution for SoC balancing in islanded microgrids. By enabling decentralized training and execution, the system overcomes the limitations of centralized controllers. Future work will explore grid-connected modes, energy trading dynamics, and scaling the system to hundreds of agents to further validate the consensus algorithm's performance.

Summary updated

BibTeX
@article{xiong2024decentralized,
  author = {Zheng Xiong and Biao Luo and Bing-Chuan Wang and Xiaodong Xu and Xiaodong Liu and Tingwen Huang},
  title = {{Decentralized Multi-agent Reinforcement Learning based State-of-Charge Balancing Strategy for Distributed Energy Storage System}},
  journal = {IEEE Trans. Industrial Informatics},
  year = {2024},
  doi = {10.1109/tii.2024.3424522},
}
Figure