Reinforcement Learning for Optimal Frequency Control: A Lyapunov Approach

Wenqi Cui, Baosen Zhang

Abstract: As more inverter-connected renewable resources are integrated into the grid, frequency stability may degrade because of the reduction in mechanical inertia and damping. A common approach to mitigate this degradation in performance is to use the power electronic interfaces of the renewable resources for primary frequency control. Since inverter-connected resources can realize almost arbitrary responses to frequency changes, they are not limited to reproducing the linear droop behaviors. To fully leverage their capabilities, reinforcement learning (RL) has emerged as a popular method to design nonlinear controllers to optimize a host of objective functions. Because both inverter-connected resources and synchronous generators would be a significant part of the grid in the near and intermediate future, the learned controller of the former should be stabilizing with respect to the nonlinear dynamics of the latter. To overcome this challenge, we explicitly engineer the structure of neural network-based controllers such that they guarantee system stability by construction, through the use of a Lyapunov function. A recurrent neural network architecture is used to efficiently train the controllers. The resulting controllers only use local information and outperform optimal linear droop as well as other state-of-the-art learning approaches.

Introduction

Traditional power grids rely on the physical inertia of massive spinning generators to dampen frequency fluctuations. As the energy landscape shifts toward "inverter-connected" renewables like solar and wind, this natural inertia disappears, making the grid more volatile and prone to blackouts. While inverters are highly flexible and can be programmed with complex control laws, designing these laws is difficult. Engineers typically rely on simple "linear droop" controls to guarantee stability. Reinforcement Learning (RL) offers a path toward superior nonlinear control, but standard RL often treats stability as a "soft" goal via penalties rather than a "hard" requirement. In critical infrastructure like a power grid, an RL agent that is only "usually stable" represents a significant liability.

Methodology

This research bridges the gap between deep learning and control theory through three primary innovations designed to ensure reliability by design.

Overview of the proposed framework, integrating the power grid dynamics with a stability-guaranteed neural network controller and an RNN-based training loop.
  1. Stability by Construction: Using Lyapunov stability theory, the authors prove that if a decentralized controller is a monotonically increasing function that passes through the origin (where zero frequency deviation results in zero control action), the entire nonlinear power system is guaranteed to be locally exponentially stable.
  2. Stacked-ReLU Neural Networks: To enforce this monotonicity, the team designed a specific neural network architecture. By constraining the weights and biases of a "stacked" ReLU (Rectified Linear Unit) network, they ensure the output always increases with the input, satisfying the stability requirement regardless of the network's learned parameters.
  3. RNN-based Training: To train the controller efficiently, the power system's physics (the "swing equations") are treated as a cell within a Recurrent Neural Network (RNN). This allows the use of standard deep learning backpropagation to optimize the controller over a time horizon, making training significantly faster than model-free RL approaches.

Key Results

The approach was evaluated using the IEEE New England 39-bus (NE39) benchmark and the Power System Toolbox (PST). The results demonstrate that the proposed RNN-Lyapunov method provides a robust solution to the stability-performance trade-off.

  • Guaranteed Stability: In simulations, standard RL without Lyapunov constraints often led to unstable, divergent oscillations. In contrast, the RNN-Lyapunov method remained perfectly stable across all test cases.
  • Performance Gains: The learned nonlinear controller reduced the average loss (a combination of frequency deviation and control effort) by approximately 11.4% compared to the best possible linear droop controller.
  • Training Efficiency: The physics-informed RNN-based training approach was roughly 74% faster than standard policy gradient methods like REINFORCE.
  • Robustness: The controllers maintained performance even when tested on high-fidelity, 6th-order generator models that were more complex than the simplified models used for the initial mathematical proofs.

Conclusion

By building stability constraints directly into the neural network's architecture, this framework eliminates the risk of unstable trajectories during deployment. While the current proof guarantees local exponential stability, future work will focus on extending these guarantees to multi-area power systems and exploring interactions with other grid assets like high-voltage DC (HVDC) lines. This research demonstrates that enforcing simple local properties, such as monotonicity, can solve complex global stability problems in modern power grids.

Summary updated

Abstract

As more inverter-connected renewable resources are integrated into the grid, frequency stability may degrade because of the reduction in mechanical inertia and damping. A common approach to mitigate this degradation in performance is to use the power electronic interfaces of the renewable resources for primary frequency control. Since inverter-connected resources can realize almost arbitrary responses to frequency changes, they are not limited to reproducing the linear droop behaviors. To fully leverage their capabilities, reinforcement learning (RL) has emerged as a popular method to design nonlinear controllers to optimize a host of objective functions. Because both inverter-connected resources and synchronous generators would be a significant part of the grid in the near and intermediate future, the learned controller of the former should be stabilizing with respect to the nonlinear dynamics of the latter. To overcome this challenge, we explicitly engineer the structure of neural network-based controllers such that they guarantee system stability by construction, through the use of a Lyapunov function. A recurrent neural network architecture is used to efficiently train the controllers. The resulting controllers only use local information and outperform optimal linear droop as well as other state-of-the-art learning approaches.
BibTeX
@article{cui2022reinforcement,
  author = {Wenqi Cui and Baosen Zhang},
  title = {{Reinforcement Learning for Optimal Frequency Control: A Lyapunov Approach}},
  journal = {IEEE Transactions on Power Systems},
  year = {2022},
  doi = {10.1109/tpwrs.2022.3176525},
}
Figure