Typical Power Grid Operation Mode Generation Based on Reinforcement Learning and Deep Belief Network

Zirui Wang, Bowen Zhou, Chen Lv, Hongming Yang, Quan Ma, Zhao Yang, Yong CuiSustainability 2023 · DOI

TL;DR

This paper introduces a novel method for generating typical power grid operation modes using Q-learning for intelligent mode generation and Deep Belief Networks (DBN) for rapid evaluation. This approach aims to overcome the limitations of manual experience and improve the efficiency and accuracy of operation mode calculations in increasingly complex power grids.

What Problem Does This Paper Address?

Modern power grids are becoming more complex due to the integration of renewable energy sources, leading to increased operational uncertainty. Traditional methods for determining optimal power grid operation modes rely heavily on manual experience, which is becoming inefficient, time-consuming, and prone to errors. The sheer volume of potential operation modes and the intricate interdependencies make it challenging to identify the most suitable ones for ensuring grid stability and economic efficiency. This paper addresses the need for an automated, efficient, and accurate system for generating and evaluating these critical operation modes.

Methods

The proposed method integrates several key techniques:

  • Q-Learning for Operation Mode Generation: Q-learning, a reinforcement learning algorithm, is employed to intelligently generate operation modes. It treats the power grid as an environment and a system operator as an agent that learns to adjust generator switching to achieve optimal power flow convergence and meet operational constraints. The algorithm iteratively updates its strategy based on rewards received from the environment.
  • DBSCAN Clustering: After generating a large set of operation modes, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is used to group similar modes into distinct operation mode sets. This helps in organizing the generated modes and identifying representative clusters.
  • AHP-EWM for Index Integration: A comprehensive evaluation system is developed using indexes from three perspectives: steady-state security, transient-state security, and economy. The Analytical Hierarchy Process (AHP) and Entropy Weight Method (EWM) are combined (AHP-EWM) to assign weights to these multi-dimensional indexes, creating a single composite value for each operation mode.
  • Deep Belief Network (DBN) for Rapid Evaluation: A DBN is utilized to build a rapid operation mode evaluation model. Instead of recalculating all indexes for each mode, the DBN learns the correlation between operation mode features and their comprehensive evaluation values. This allows for quick and accurate prediction of the composite values for new operation modes.

Datasets

The study uses simulation examples based on the actual historical operation modes of the Northeast China Power Grid. This grid comprises four provincial-level regions with 2128 nodes, 531 generators, and 816 load buses. The paper simulates future load growth by increasing load levels by 5% year-on-year across low valley, flat waist, and peak scenarios.

Key Insights and Ideas

  • Reinforcement Learning for Grid Control: Applying Q-learning to power grid operation mode generation demonstrates its effectiveness in learning complex control strategies for dynamic systems, moving beyond traditional rule-based or experience-driven approaches.
  • Clustering for Mode Organization: DBSCAN proves useful in organizing the vast number of generated operation modes into meaningful sets, making the subsequent evaluation and selection process more manageable.
  • Multi-dimensional Evaluation is Crucial: The paper emphasizes the importance of a comprehensive evaluation system that considers steady-state, transient-state, and economic factors, highlighting that transient-state security is particularly critical for real-world grid operations.
  • DBN for Efficient Prediction: The use of DBN for rapid evaluation significantly reduces the computational burden compared to traditional methods, enabling faster decision-making for system operators.

Key Results

  • Q-Learning Performance: The Q-learning algorithm converged faster and achieved higher cumulative reward values (45) compared to the SARSA algorithm (38) in generating operation modes.
  • Mode Generation: The method successfully generated typical operation modes across different load levels, showing variations in generator output across regions.
  • Clustering Results: The number of operation mode sets decreased as the load level increased, indicating that higher demand leads to more consolidated operational strategies.
  • Evaluation Model Accuracy: The DBN-based evaluation model demonstrated strong prediction abilities, outperforming traditional machine learning algorithms like KNN, SVR, and XGBoost in terms of RMSE and MAPE. For instance, the DBN achieved an RMSE of 0.506 and MAPE of 0.458, significantly lower than XGBoost's RMSE of 1.354 and MAPE of 1.068.
  • Efficiency Improvement: The proposed method significantly improves the efficiency of operation mode calculation, reducing reliance on manual labor and material resources.

Limitations and Future Work

  • Power Grid Structure: The current method generates operation modes without considering changes in the power grid structure itself. Future work could incorporate dynamic grid topology into the generation process.
  • Concentrated vs. Distributed High-Value Modes: The paper notes that if high-value operation modes are concentrated within a single set, selecting the highest is straightforward. However, if they are distributed across multiple sets, system operators need to make more complex judgments. Future research could focus on analyzing correlation constraints among these distributed high-value modes to better identify typical operation modes.

Summary updated

Abstract

With the continuous expansion of power grids and the gradual increase in operational uncertainty, it is progressively challenging to meet the capacity requirements for power grid development based on manual experience. In order to further improve the efficiency of the operation mode calculation, reduce the consumption of manpower and material resources, and consider the sustainability of energy development, this paper proposes a typical power grid operation mode generation method based on Q-learning and the deep belief network (DBN) for the first time. Firstly, the operation modes of different generator combinations located in different regions are obtained through Q-learning intelligent generation. Subsequently, the generated operation modes are clustered as different operation mode sets according to the data characteristics. Furthermore, comprehensive evaluation indexes are proposed from the perspectives of the steady state, transient state, and the economy. These multi-dimensional indexes are integrated via the analytical hierarchy process–entropy weight method (AHP-EWM) to enhance the comprehensibility of the evaluation system. Finally, DBN is introduced to construct a rapid operation mode evaluation model to realize the evaluation of operation mode sets, and typical operation mode sets are obtained accordingly. In this way, the system calculator only needs to compare the composite values to obtain the typical operation modes. The proposed method is validated by the Northeast Power Grid in China. The experimental results show that the proposed method can quickly generate typical power grid operation modes according to actual demand and greatly improve the efficiency of operation mode calculation.
BibTeX
@inproceedings{wang2023typical,
  author = {Zirui Wang and Bowen Zhou and Chen Lv and Hongming Yang and Quan Ma and Zhao Yang and Yong Cui},
  title = {{Typical Power Grid Operation Mode Generation Based on Reinforcement Learning and Deep Belief Network}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su152014844},
}
Figure