PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks

Anandsingh Chauhan, Mayank Baranwal, Ansuma BasumataryAAAI 2023 · DOI

TL;DR

This paper introduces PowRL, a novel reinforcement learning (RL) framework designed to enhance the robustness of power grids against uncertainties like renewable energy fluctuations and adversarial attacks. By combining a reduced action space, heuristic-guided decision-making, and a Proximal Policy Optimization (PPO) agent, PowRL achieves state-of-the-art performance on challenging power grid management benchmarks.

What Problem Does This Paper Address?

Modern power grids are increasingly complex due to the integration of renewable energy sources (like solar and wind) and the rise of electric vehicles, leading to unpredictable generation and demand. This variability, coupled with the threat of adversarial cyberattacks, makes maintaining grid stability and ensuring reliable power delivery a significant challenge. Traditional control strategies struggle to cope with the sheer scale and dynamism of these issues. Specifically, the problem of managing power networks involves making sequential decisions in a vast state-action space, where even a moderately sized network can present tens of thousands of possible actions. This complexity often exceeds human cognitive capabilities, leading grid operators to rely on experience or limited heuristics. The paper aims to develop an AI-driven solution that can extract maximum resiliency from current grid infrastructure to ensure smooth power delivery and provide ancillary services.

Methods

PowRL employs a multi-faceted approach:

  • Reduced Action Space: The paper addresses the combinatorial explosion of possible actions in power grid control. By analyzing the IEEE-118 network, they identified a significantly reduced set of 240 critical topological actions that have the most impact, making the learning process more tractable.
  • Heuristic-Guided Topological Actions: PowRL augments its RL agent with baseline heuristics. These include disengaging the RL agent during stable operation, manually disconnecting overloaded lines with a plan for reconnection, restoring configurations to their original state after contingencies, and avoiding actions that would bifurcate the network.
  • Proximal Policy Optimization (PPO)-Guided Reinforcement Learning Framework: The core of PowRL is an on-policy PPO agent trained using a prioritized replay buffer. The agent's state includes detailed information about the grid's time, generator and load features, line status and flow, topological configuration, and cooldown timers. The reward function penalizes overflows beyond a safety threshold and rewards surviving the entire episode, while penalizing premature termination due to illegal actions or blackouts.
  • Optimal Action Selection: A key innovation is the structured optimal action selection process. PowRL first simulates a "do-nothing" action. If this action keeps the grid within safety thresholds, it's either implemented or a recovery action is taken if the current state deviates from the initial configuration. If the "do-nothing" action leads to an overflow, the RL agent's recommended action is then considered, potentially combined with heuristic actions.

Datasets

The paper utilizes datasets from the L2RPN (Learning to Run a Power Network) challenge:

  • L2RPN NeurIPS 2020 Challenge Datasets:
    • Training Data: A small dataset from the Robustness track, comprising 48 years of power grid data at 5-minute intervals, based on the reduced IEEE-118 system.
  • Offline Test Dataset: Provided with the starting kit for evaluation.
  • Online Test Dataset: A hidden dataset used for competition evaluation.
  • L2RPN WCCI 2020 Challenge Dataset: An offline evaluation dataset with 10 different 3-day episodic scenarios.

Key Insights and Ideas

  • The Power of Reduced Action Spaces: Demonstrates that focusing on a carefully selected subset of critical topological actions can be more effective and efficient than exploring a vast, unmanageable action space.
  • Synergy of Heuristics and RL: Combining established heuristic rules with a learned RL policy leads to more robust and reliable decision-making, especially in critical situations.
  • Diversified Action Sequences: PowRL's analysis reveals that its agent learns to utilize a wider range of substations and develops more diverse action sequences compared to other agents, suggesting a deeper understanding of network dynamics.
  • Targeted Attack Mitigation: The framework shows an ability to identify and learn optimal action sequences to protect the network against specific, targeted adversarial attacks.

Key Results

PowRL achieved state-of-the-art performance across multiple L2RPN challenge datasets:

  • L2RPN NeurIPS 2020 Online: PowRL, along with the rl_agent, survived the highest number of scenarios (16 out of 24). It successfully navigated challenging scenarios like dec_16.2 and aug_02.1, outperforming other baselines.
  • L2RPN NeurIPS 2020 Offline: While lujixiang survived the most scenarios (12), PowRL's performance indicated lower operational costs and fewer blackouts.
  • L2RPN WCCI 2020 Offline: PowRL and rl_agent successfully survived all scenarios, demonstrating strong performance in congestion management.
  • Efficiency: PowRL achieved the least runtime among RL agents on the NeurIPS Online challenge.
  • Action Diversity: PowRL utilized a more diverse set of actions across substations (25) compared to other agents (e.g., rl_agent used 18).

Limitations and Future Work

  • Limitations:
    • The paper notes that some scenarios in the NeurIPS 2020 challenge required flexibilities like voltage control and load shedding, which were not available in the environment.
  • The lujixiang agent's source code was reported as broken, limiting a full comparison on online data.
  • The binbinchen agent showed signs of overfitting on the WCCI dataset.
  • Future Work:
    • The paper suggests exploring more advanced control actions such as voltage control and load shedding to further enhance grid resilience.
  • Investigating how to integrate these additional control mechanisms into the RL framework would be a logical next step.

Summary updated

BibTeX
@inproceedings{chauhan2023powrl,
  author = {Anandsingh Chauhan and Mayank Baranwal and Ansuma Basumatary},
  title = {{PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26724},
}
Figure