Secondary Voltage Collaborative Control of Distributed Energy System via Multi-Agent Reinforcement Learning

Tianhao Wang, Shiqian Ma, Na Xu, Tianchun Xiang, Xiaoyun Han, Chaoxu Mu, Jin YaoEnergies 2022 · DOI

TL;DR

This paper introduces a novel voltage control strategy for distributed energy systems using a multi-agent reinforcement learning approach enhanced with an attention mechanism. This method allows distributed energy nodes to collaboratively restore voltage stability more efficiently by intelligently prioritizing communication, leading to improved performance compared to existing methods.

What Problem Does This Paper Address?

The increasing integration of renewable energy sources into power grids, while beneficial, introduces challenges related to grid stability, particularly voltage fluctuations. Distributed energy systems (DES), which are decentralized and closer to consumers, need robust control mechanisms to manage these variations. Traditional control methods often struggle with the dynamic and unpredictable nature of renewables. Existing multi-agent reinforcement learning (MARL) approaches for DES control can be communication-intensive and may not efficiently filter critical information, leading to suboptimal performance. This paper aims to address the need for a more efficient and effective secondary voltage control strategy in DES that can adapt to these challenges.

Methods

The core of the proposed solution is the Multi-Agent Advantage Actor-Critic (MA2C) algorithm, a MARL framework where multiple agents learn to cooperate. To enhance this, the authors introduce an Attentional Actor-Critic Message Processor (AACMP). This attention mechanism allows each agent (representing an energy node) to selectively focus on and process the most relevant information from its neighbors, rather than treating all communication equally. This is crucial for reducing communication overhead and improving the efficiency of decision-making in a decentralized system.

The control strategy operates under a centralized training and decentralized execution paradigm. During training, the system learns optimal policies in a simulated environment. Once trained, each agent can operate autonomously in the real system, making decisions based on its local observations and the filtered information from its neighbors. The MA2C algorithm uses an actor network to decide on actions (voltage setpoints) and a critic network to evaluate those actions, aiming to maximize cumulative rewards. The attention mechanism specifically refines how agents share and interpret messages, leading to a more coordinated response.

Datasets

Not discussed in the paper. The authors built a simulated distributed energy system for their experiments.

Key Insights and Ideas

  • Intelligent Communication is Key: The paper highlights that in complex, interconnected systems like DES, simply exchanging all available information isn't efficient. An "attention" mechanism that prioritizes critical messages significantly improves coordination and reduces communication burden.
  • MARL for Decentralized Control: Reinforcement learning, particularly multi-agent variants, is a powerful tool for achieving cooperative control in decentralized systems where agents must learn to work together without a central commander.
  • Balancing Centralized Learning with Decentralized Action: The centralized training/decentralized execution framework is effective for learning complex coordination strategies while allowing for real-time, autonomous operation of individual energy nodes.
  • Voltage Stability is Paramount: The reward function is designed to directly penalize voltage deviations, ensuring that the learning process prioritizes maintaining voltage within acceptable limits.

Key Results

The proposed MA2C-Attention algorithm demonstrated superior performance compared to a standard MA2C algorithm and an Independent Advantage Actor-Critic (IA2C) algorithm.

  • Faster Convergence and Higher Rewards: The MA2C-Attention algorithm achieved a higher average episode reward and converged faster during training, as visualized in Figure 5.
  • Improved Voltage Control: In random tests, the MA2C-Attention algorithm consistently resulted in voltage values closer to the reference setpoint for all six distributed generators compared to MA2C, IA2C, and traditional droop control (Table 2 and Figure 6). For instance, the average voltage for Agent 1 was 1.00975 pu with MA2C-Attention, compared to 1.00467 pu for MA2C and 0.97759 pu for IA2C.
  • Reduced Voltage Fluctuations: The proposed method effectively promoted node voltage recovery to the reference value and maintained relatively stable voltage fluctuations within the allowable range.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Simplified System Model: The simulated distributed energy system is a simplified representation and does not fully capture the complexity of an actual power system.
  • Frequency Offset Ignored: While voltage offset was considered, the influence of frequency offset on system stability was not explicitly addressed in this study.
  • Future Work:
    • Building a more realistic simulation environment by incorporating actual operating system data.
  • Designing a three-level control scheme for the distributed energy system.
  • Developing an objective function tailored to user needs for optimal system operation.
  • Designing a dispatch scheme using MARL to provide optimal power generation setpoints for each distributed energy source.

Summary updated

Abstract

In this paper, a new voltage cooperative control strategy for a distributed power generation system is proposed based on the multi-agent advantage actor-critic (MA2C) algorithm, which realizes flexible management and effective control of distributed energy. The attentional actor-critic message processor (AACMP) is extended into the MA2C method to select the important messages from all communication messages adaptively and process important messages efficiently. The cooperative control strategy trained by centralized training and decentralized execution frame will take over the responsibility of the secondary control level for voltage restoration in a distributed manner. The introduction of the attention mechanism reduces the amount of information exchanged and the requirements of the communication network. Finally, a distributed system with six energy nodes is used to verify the effectiveness of the proposed control strategy.
BibTeX
@inproceedings{wang2022secondary,
  author = {Tianhao Wang and Shiqian Ma and Na Xu and Tianchun Xiang and Xiaoyun Han and Chaoxu Mu and Jin Yao},
  title = {{Secondary Voltage Collaborative Control of Distributed Energy System via Multi-Agent Reinforcement Learning}},
  booktitle = {Energies},
  year = {2022},
  doi = {10.3390/en15197047},
}
Figure