Abstract
This paper introduces Dec-MARL, a fully decentralized reinforcement learning framework designed to balance the State-of-Charge (SoC) across distributed battery systems in a microgrid. By combining multi-agent deep reinforcement learning with a novel "counterfactual demand balance" algorithm, the system ensures that batteries of varying capacities drain and charge at rates that keep them synchronized, preventing system overloads without requiring a central controller.
Introduction
In modern microgrids, Distributed Energy Storage Systems (DESS) are vital for smoothing out the intermittent nature of renewable energy. However, these batteries often have different capacities and initial charge levels. Poor management leads to rapid depletion of some units, forcing others to compensate and causing potential system failures. Traditional centralized control methods suffer from single points of failure and poor scalability. While Multi-Agent Reinforcement Learning (MARL) offers a solution, most existing versions rely on Centralized Training, Decentralized Execution (CTDE). This paper proposes a Decentralized Training, Decentralized Execution (DTDE) approach, where agents learn and act entirely based on local communication.
Methodology
The proposed Dec-MARL framework relies on three primary pillars:
- Observation Expansion via Consensus: Agents use a first-order average consensus algorithm to estimate global average SoC and total power demand by communicating with immediate neighbors.
- Counterfactual Demand Balance (CDB) Algorithm: This core innovation manages the global constraint that total power output must match total demand. Unlike standard factual clipping, the counterfactual approach allows for better exploration of the action space, accelerating learning.
- Deep Deterministic Policy Gradient (DDPG): Each battery unit operates as an independent RL agent using a DDPG architecture to decide its power output based on expanded observations.
Experimental Setup
The study utilizes a custom-built simulation environment of a microgrid with 5 Energy Storage Units (ESUs) with capacities ranging from 700kWh to 1800kWh. Training used a sinusoidal power demand to simulate daily fluctuations, while performance testing used linear ramp-up and constant demand profiles to evaluate adaptation to changing conditions.
Key Results and Discussion
The Dec-MARL system demonstrated successful convergence after approximately 6,000 training episodes. Key findings include:
- Robust Decentralization: Using the Metropolis-Hastings weight matrix for communication, the system avoids a central controller, making it resilient to failures or the addition of new units.
- Effective Balancing: Dec-MARL significantly reduced SoC variance compared to proportional allocation baselines, synchronizing all five batteries to a similar SoC (roughly 0.8).
- Constraint Satisfaction: The CDB algorithm effectively managed global power demand constraints locally, proving that independent agents can satisfy global requirements through counterfactual exploration.
Conclusion
The Dec-MARL framework provides a scalable and robust solution for SoC balancing in islanded microgrids. By enabling decentralized training and execution, the system overcomes the limitations of centralized controllers. Future work will explore grid-connected modes, energy trading dynamics, and scaling the system to hundreds of agents to further validate the consensus algorithm's performance.
Summary updated