Distributed Energy Trading and Scheduling among Microgrids via Multiagent Reinforcement Learning

Guanyu Gao, Yonggang Wen, Xiaohu Wu, Ran WangIEEE TNNLS 2020 · DOI · OpenReview

TL;DR

This paper tackles the complex challenge of optimizing energy trading and scheduling for interconnected microgrids. They propose a novel multi-agent deep reinforcement learning (MADRL) approach that allows each microgrid to learn optimal decentralized control policies, leading to significant cost reductions and improved overall rewards compared to existing methods.

What Problem Does This Paper Address?

The increasing adoption of renewable energy sources is leading to the proliferation of microgrids, which are essentially localized energy grids that can operate independently or connect to the main power grid. These microgrids can generate their own electricity, store it, and trade surplus energy with other microgrids or the broader energy market. However, optimizing the energy trading and scheduling decisions for each microgrid is a significant challenge. This complexity arises from:

  • Interdependence: A microgrid's trading decisions are influenced by the actions of other microgrids.
  • Coupling: Trading and local scheduling (e.g., battery charging/discharging, load management) are intrinsically linked.
  • Uncertainty: Renewable energy generation (solar, wind) and energy consumption are inherently variable and difficult to predict precisely.
  • Autonomy: Maintaining the independent decision-making capability of each microgrid is crucial.

Traditional modeling-based approaches often struggle with these complexities, requiring precise future system dynamics that are hard to obtain. This paper aims to address this by developing a learning-based approach that allows microgrids to learn optimal control policies through direct interaction with their environment.

Methods

The core of this paper is the proposal of a Multi-Agent Deep Reinforcement Learning (MADRL) approach, specifically leveraging the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. Here's a breakdown of their methodology:

  • Microgrids as Agents: Each microgrid is modeled as an independent agent within a larger system.
  • MADDPG Framework: This algorithm is chosen because it's well-suited for:
  • Mixed Cooperative-Competitive Environments: Microgrids can compete for selling prices or collaborate to achieve mutual benefits.
  • Decentralized Execution: While training uses centralized information (for better learning), the execution phase relies solely on local information, preserving microgrid autonomy.
  • Continuous Control: The algorithm can handle continuous actions like trading prices, quantities, and charging/discharging rates.
  • Actor-Critic Architecture: Each agent has an actor network (to decide actions) and a critic network (to evaluate those actions). The critic network, during training, receives states and actions from all agents to provide a comprehensive evaluation.
  • State Representation: The state for each microgrid agent includes its local information (battery level, historical generation/load) and public information (wholesale market price, historical market clearing price).
  • Action Space: Actions include setting selling/buying prices, trading quantities, and battery charging/discharging amounts.
  • Reward Function: The reward for each microgrid is defined as the sum of its revenues (selling energy) minus its costs (buying energy from wholesale and hour-ahead markets) and any penalties for contract violations.
  • Double Auction Mechanism: A double auction mechanism is used for hour-ahead market trading, where microgrids submit bids and offers, and a clearing price and quantity are determined.

Datasets

The authors utilized real-world datasets for their simulations:

  • Load Profiles: Scaled historical electricity system demand data for Singapore.
  • Renewable Energy Generation: Solar radiation data from Solcast for Singapore.
  • Wholesale Market Price: A fixed price of 22.79 cents/kWh (Singapore).
  • Hour-Ahead Market Price Range: 15.0 to 22.79 cents/kWh.

The paper does not introduce a new benchmark dataset but rather uses these real-world data sources to create a realistic simulation environment.

Key Insights and Ideas

  • MADRL for Decentralized Optimization: The paper demonstrates that MADRL is a powerful tool for learning optimal decentralized control policies in complex, multi-agent energy systems, effectively balancing cooperation and competition.
  • Autonomy Preserved: The decentralized execution aspect of MADDPG is crucial for maintaining the independence of individual microgrids, a key requirement in practical microgrid operations.
  • Learning Beyond Modeling: By directly interacting with the environment, the MADRL approach bypasses the need for precise, often unattainable, system models, making it more robust to uncertainties.
  • Hour-Ahead Market Advantage: The results highlight that trading on the hour-ahead market, even with its own complexities, can be significantly more cost-effective than relying solely on the wholesale market, especially for microgrids with lower renewable generation capacity.
  • Microgrid Characteristics Matter: The performance and strategies of microgrids are heavily influenced by their renewable generation capacity and load profiles, leading to distinct behaviors (e.g., sellers vs. buyers).

Key Results

  • Significant Cost Reduction: The proposed MADRL method consistently reduced the overall costs for all microgrids compared to baseline methods (Isolated and single-agent DDPG).
  • Improved Overall Reward: Microgrids achieved higher overall rewards (revenue minus costs) using the proposed method.
  • Higher Trading Success: Microgrids using the proposed method exhibited a higher successful trading ratio and a larger average trading quantity on the hour-ahead market.
  • Converged Policies: The agents' rewards converged during training, indicating that the MADRL approach successfully found stable, optimal policies for each microgrid. Microgrids with higher renewable generation (Microgrid 1 & 2) achieved positive converged rewards due to profitable selling, while others (Microgrid 3 & 4) still saw significant cost savings and reward improvements.

Limitations and Future Work

  • Exploration Noise: The paper mentions using an Ornstein-Uhlenbeck process for exploration noise, which is a standard technique but can sometimes be tuned for better performance.
  • Simulation Environment: While using real-world data, the performance is evaluated within a simulated environment. Real-world deployment might introduce additional complexities.
  • Scalability: The paper focuses on four microgrids. While MADDPG is designed for multi-agent systems, the computational complexity can increase significantly with a larger number of microgrids.
  • Future Work: The authors suggest exploring more complex scenarios, such as incorporating energy storage degradation, different types of renewable energy sources, and more sophisticated market mechanisms. They also mention the potential for further research into the cooperative and competitive dynamics between microgrids.

Summary updated

Abstract

The development of renewable energy generation empowers microgrids to generate electricity to supply itself and to trade the surplus on energy markets. To minimize the overall cost, a microgrid must determine how to schedule its energy resources and electrical loads and how to trade with others. The control decisions are influenced by various factors, such as energy storage, renewable energy yield, electrical load, and competition from other microgrids. Making the optimal control decision is challenging, due to the complexity of the interconnected microgrids, the uncertainty of renewable energy generation and consumption, and the interplay among microgrids. The previous works mainly adopted the modeling-based approaches for deriving the control decision, yet they relied on the precise information of future system dynamics, which can be hard to obtain in a complex environment. This work provides a new perspective of obtaining the optimal control policy for distributed energy trading and scheduling by directly interacting with the environment, and proposes a multiagent deep reinforcement learning approach for learning the optimal control policy. Each microgrid is modeled as an agent, and different agents learn collaboratively for maximizing their rewards. The agent of each microgrid can make the local scheduling decision without knowing others' information, which can well maintain the autonomy of each microgrid. We evaluate the performances of our proposed method using real-world datasets. The experimental results show that our method can significantly reduce the cost of the microgrids compared with the baseline methods.
BibTeX
@article{gao2020distributed,
  author = {Guanyu Gao and Yonggang Wen and Xiaohu Wu and Ran Wang},
  title = {{Distributed Energy Trading and Scheduling among Microgrids via Multiagent Reinforcement Learning}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2020},
  doi = {10.1109/tnnls.2022.3170070},
}
Figure