MAHTM: A Multi-Agent Framework for Hierarchical Transactive Microgrids

Nicolas Cuadrado, Roberto Gutierrez, Yongli Zhu, Martin Takác

TL;DR

This paper introduces MAHTM, a novel multi-agent reinforcement learning framework for managing energy transactions in microgrids. By employing a hierarchical structure and optimizing for both cost and carbon footprint, MAHTM aims to facilitate the integration of renewable energy sources and foster local energy markets.

What Problem Does This Paper Address?

Integrating variable renewable energy sources like solar and wind into existing power grids presents a significant challenge for system operators. The inherent stochasticity of these sources makes it difficult to balance energy availability, cost affordability, and environmental impact (carbon emissions). Traditional grid management struggles to cope with these fluctuations and optimize energy trade-offs. This paper addresses this by proposing a system for "transactive microgrids," which are localized energy networks that can intelligently manage energy production, consumption, and trading among participants. The goal is to create an efficient and environmentally friendly energy ecosystem within these microgrids, minimizing human intervention and allowing for the sale of surplus energy.

Methods

The core of the proposed solution is the MAHTM (Multi-Agent Hierarchical Framework for Transactive Microgrids). This framework utilizes a three-layer hierarchical reinforcement learning (RL) architecture:

  • Layer 1: Household Level: Agents at this level represent individual households. They are categorized into "passive consumers," "passive prosumers" (with PV), "active prosumers" (with PV and batteries), and "active consumers" (with storage). Their primary objective is to minimize their total energy cost by deciding how to charge/discharge batteries and manage their demand.
  • Layer 2: Microgrid Level: Each microgrid has an agent responsible for setting local energy prices. Its objective is to maximize the utilization of local energy by defining pricing policies for internal transactions. This layer balances energy imports and exports between microgrids and the distributor.
  • Layer 3: Distributor Level: This top-level agent shapes the overall load across multiple microgrids. It sets buy and sell prices between microgrids and the distributor, aiming to minimize the carbon footprint.

The framework employs a Multi-Agent Reinforcement Learning (MARL) approach, specifically using an Advantage Actor-Critic (A2C) algorithm. This is a standard MARL technique where each agent has its own actor (policy) and critic (value function). The novelty lies in the hierarchical structure and the specific objectives assigned to each layer, which collectively aim to optimize for cost and carbon emissions.

Datasets

The paper describes an environment configured using OpenAI Gym. The dataset used for training, validation, and testing was synthetically generated based on real-life data. Key features of the dataset include:

  • Demand Profiles: Three distinct profiles (family, teenager, business) with varying peak times and noise.
  • PV Generation: Stochastic generation based on solar panels.
  • Battery Characteristics: Including random initial state of charge, capacity, efficiency, and charge/discharge limits.
  • Grid Energy Cost and Carbon Footprint: Modeled using two sources, nuclear (stable, lower cost/emissions) and gas (more expensive, higher emissions).

The authors note that their environment allows for enabling/disabling stochasticity, differentiating it from some prior work.

Key Insights and Ideas

  • Hierarchical Optimization is Key: The multi-layered approach allows for specialized optimization at each level (household cost, microgrid local energy use, distributor carbon footprint), which collectively leads to a more holistic system optimization than a flat structure.
  • Balancing Cost and Carbon: The framework explicitly aims to minimize both energy costs for consumers and the overall carbon footprint, demonstrating that these objectives can be pursued simultaneously within a transactive microgrid.
  • MARL for Grid Management: The paper showcases the effectiveness of MARL in handling the complex, dynamic, and multi-agent nature of microgrid energy management, where individual agents must learn to cooperate and compete.
  • Stochasticity Handling: The ability to incorporate and test with noisy data (simulating real-world weather variations) is crucial for practical microgrid applications.

Key Results

The paper presents performance metrics comparing their MAHTM framework (using A2C) against a linear solver (CVXPY) and another MARL algorithm (COMA).

  • Performance: MAHTM achieved results "very close to the optimal" within a reasonable training time.
  • Training Speed and Simplicity: MAHTM is highlighted as being simpler and faster to train compared to COMA, which is sensitive to hyperparameter tuning.
  • Quantitative Comparison (Table 2):
    • Train Reward: MAHTM (-0.993) is better than CVXPY (-0.915) and COMA (-1.3).
  • Train Price Score: MAHTM (-0.097) is better than CVXPY (-0.103) but worse than COMA (0.35).
  • Train Emission Score: MAHTM (-0.1522) is better than CVXPY (-0.223) and COMA (0.35).
  • Test Price Score: MAHTM (-0.064) is better than CVXPY (-0.0889) and COMA (0.0625).
  • Test Emission Score: MAHTM (-0.097) is better than CVXPY (-0.19) and COMA (0.0625).
  • Qualitative Results (Figures 2-5): The figures demonstrate the learned behaviors of the agents, showing how they manage SOC, net energy, and respond to price and emission factors across different scenarios (with/without noise, different demand profiles).

Limitations and Future Work

  • Limitations:
    • The paper mentions that in the current version, there is no battery cell price, which is a planned parameter for future incorporation to add more dynamics.
  • The distributor layer assumes only one distributor, and the energy consumed within or between microgrids has negligible carbon impact, which might be an oversimplification.
  • The performance comparison with COMA shows mixed results, with COMA outperforming MAHTM on some metrics (e.g., Train Price Score).
  • Future Work:
    • Incorporating battery cell prices.
  • Exploring more complex distributor models.
  • Further investigation into the dynamics of local energy markets.
  • The authors suggest their framework can help develop local renewable energy markets and foster consumer engagement.

Summary updated

Abstract

Integrating variable renewable energy into the grid has posed challenges to system operators in achieving optimal trade-offs among energy availability, cost affordability, and pollution controllability. This paper proposes a multi-agent reinforcement learning framework for managing energy transactions in microgrids. The framework addresses the challenges above: it seeks to optimize the usage of available resources by minimizing the carbon footprint while benefiting all stakeholders. The proposed architecture consists of three layers of agents, each pursuing different objectives. The first layer, comprised of prosumers and consumers, minimizes the total energy cost. The other two layers control the energy price to decrease the carbon impact while balancing the consumption and production of both renewable and conventional energy. This framework also takes into account fluctuations in energy demand and supply.
BibTeX
@misc{cuadrado2023mahtm,
  author = {Nicolas Cuadrado and Roberto Gutierrez and Yongli Zhu and Martin Takác},
  title = {{MAHTM: A Multi-Agent Framework for Hierarchical Transactive Microgrids}},
  year = {2023},
  eprint = {2303.08447},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.08447},
}
Figure