Model-Based Reinforcement Learning Method for Microgrid Optimization Scheduling

Jinke Yao, Jiachen Xu, Ning Zhang, Yajuan GuanSustainability 2023 · DOI

TL;DR

This paper introduces a novel model-based reinforcement learning (RL) approach, leveraging the MuZero algorithm and Monte Carlo Tree Search (MCTS), to optimize microgrid scheduling. By learning an environmental dynamics model alongside the scheduling policy, it achieves robust and adaptive control for both islanded and grid-connected microgrids, outperforming traditional methods.

What Problem Does This Paper Address?

Microgrids, essential for integrating renewable energy sources, face significant operational instability due to the inherent uncertainty and intermittency of clean energy. Traditional optimization methods struggle with this dynamic environment, often requiring accurate environmental models that are difficult to adapt to varying microgrid configurations. Furthermore, heuristic algorithms can get stuck in local optima and lack robustness. This paper aims to develop a more adaptive and effective scheduling strategy that can handle these challenges.

Methods

The core of this work is a model-based reinforcement learning approach, specifically adapting the MuZero algorithm. MuZero is known for its success in discrete action spaces and its ability to learn a model of the environment. The authors combine MuZero with Monte Carlo Tree Search (MCTS) to enhance strategy selection.

Key methodological components include:

  • Model-Based RL with MuZero: Instead of relying solely on trial-and-error (model-free RL), this approach learns a model of the microgrid's environment dynamics. This learned model is then used by MCTS to plan future actions more effectively.
  • Monte Carlo Tree Search (MCTS): MCTS is employed to explore possible future states and actions, guided by the learned model, to find optimal scheduling decisions.
  • Three Interconnected Neural Networks: The system utilizes three main neural networks:
  • Representation Network: Encodes raw observations into a hidden state.
  • Prediction Network: Predicts the optimal strategy (policy) and expected value from the hidden state.
  • Dynamic Network: Simulates the environmental dynamics, predicting the next hidden state and reward.
  • Discretization of Action Space: To handle continuous control problems within MuZero's discrete action framework, the authors discretize the action space.
  • Joint Training: The three networks are trained jointly to simultaneously learn the environmental dynamics and the optimal scheduling policy.

Datasets

The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, it proposes and utilizes a simulated microgrid environment. This simulator is built to include key components like HVAC systems, Photovoltaic (PV) systems, and Energy Storage (ES) systems, allowing for the generation of training data and the testing of the proposed algorithm under various operational conditions.

Key Insights and Ideas

  • Model-Based RL for Microgrids is Viable: The research demonstrates that a model-based RL approach, specifically using MuZero and MCTS, can effectively tackle the complex and dynamic problem of microgrid optimization scheduling.
  • Learning the Environment Dynamics is Crucial: By learning the environmental dynamics, the algorithm can plan ahead more effectively, leading to more robust and adaptive scheduling decisions compared to model-free methods.
  • MCTS Enhances Planning: The integration of MCTS with the learned model allows for deeper exploration and more informed decision-making, especially in complex scenarios.
  • Joint Training of Networks Improves Performance: Training the representation, prediction, and dynamic networks together allows for a synergistic learning process, leading to better overall performance.
  • Islanded vs. Grid-Connected Modes: The study shows that the algorithm's effectiveness is maintained across both islanded (standalone) and grid-connected microgrid operation modes, highlighting its versatility.

Key Results

  • Effective Voltage Regulation: After 200 training steps, the algorithm successfully avoids exceeding the bus voltage limits, indicating good control over grid stability.
  • Convergence: After 800 training steps, the training results converge, with the loss values for the value and reward networks approaching zero, signifying a high degree of learning and optimization.
  • Superior Performance in Grid-Connected Mode: The study indicates that grid-connected operations, with the proposed algorithm, achieve better timely rewards and expected average optimization compared to islanded operations, suggesting enhanced stability and efficiency due to inter-microgrid interaction.
  • Policy Network Loss Remains Relatively High: While value and reward network losses converge well, the policy network's loss remains comparatively high. The authors attribute this to the large action space and the discretization process, suggesting potential for further refinement.

Limitations and Future Work

  • Discretization of Action Space: The paper acknowledges that the discretization of the action space, while necessary for MuZero, might lead to a relatively high policy network loss. This suggests that exploring more advanced methods for handling continuous action spaces or finer discretization could be beneficial.
  • Increased Complexity in Grid-Connected Mode: While grid-connected operation shows better results, the increased complexity leads to slower updates of parameter loss values for the networks. This implies that optimizing the training process for these more complex scenarios is an area for future work.
  • Generalization to Diverse Microgrid Topologies: While the paper demonstrates effectiveness across islanded and grid-connected modes, further research could explore the algorithm's generalization capabilities to a wider range of microgrid architectures and sizes.
  • Real-World Deployment: The current study is based on a simulated environment. Future work would involve testing and validating the algorithm in real-world microgrid systems to assess its practical applicability and robustness.

Summary updated

Abstract

Due to the uncertainty and randomness of clean energy, microgrid operation is often prone to instability, which requires the implementation of a robust and adaptive optimization scheduling method. In this paper, a model-based reinforcement learning algorithm is applied to the optimal scheduling problem of microgrids. During the training process, the current learned networks are used to assist Monte Carlo Tree Search (MCTS) in completing game history accumulation, and updating the learning network parameters to obtain optimal microgrid scheduling strategies and a simulated environmental dynamics model. We establish a microgrid environment simulator that includes Heating Ventilation Air Conditioning (HVAC) systems, Photovoltaic (PV) systems, and Energy Storage (ES) systems for simulation. The simulation results show that the operation of microgrids in both islanded and connected modes does not affect the training effectiveness of the algorithm. After 200 training steps, the algorithm can avoid the punishment of exceeding the red line of the bus voltage, and after 800 training steps, the training result converges and the loss values of the value and reward network converge to 0, showing good effectiveness. This proves that the algorithm proposed in this paper can be applied to the optimization scheduling problem of microgrids.
BibTeX
@inproceedings{yao2023model,
  author = {Jinke Yao and Jiachen Xu and Ning Zhang and Yajuan Guan},
  title = {{Model-Based Reinforcement Learning Method for Microgrid Optimization Scheduling}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su15129235},
}
Figure