Abstract
Transient stability of power systems is becoming increasingly important because of the growing integration of renewable resources. These resources lead to a reduction in mechanical inertia but also provide increased flexibility in frequency responses. Namely, their power electronic interfaces can implement almost arbitrary control laws. To design these controllers, reinforcement learning (RL) has emerged as a powerful method in searching for optimal non-linear control policy parameterized by neural networks. A key challenge is to enforce that a learned controller must be stabilizing. This paper proposes a Lyapunov regularized RL approach for optimal frequency control for transient stability in lossy networks. Because the lack of an analytical Lyapunov function, we learn a Lyapunov function parameterized by a neural network. The losses are specially designed with respect to the physical power system. The learned neural Lyapunov function is then utilized as a regularization to train the neural network controller by penalizing actions that violate the Lyapunov conditions. Case study shows that introducing the Lyapunov regularization enables the controller to be stabilizing and achieve smaller losses.
Introduction
As the global energy landscape transitions toward renewable energy sources (RES) like solar and wind, power grids are losing the inherent "mechanical inertia" provided by traditional synchronous generators. This reduction in inertia makes grid frequency significantly more volatile and susceptible to instability following disturbances, such as line failures—a phenomenon known as transient instability.
While the power electronics (inverters) interfacing renewables offer high flexibility for control, designing optimal control laws remains challenging. Reinforcement Learning (RL) is a promising candidate for discovering optimal non-linear controls. However, standard RL lacks safety and stability guarantees, which are critical for grid operations where instability can lead to catastrophic blackouts. Furthermore, for complex "lossy" grids with resistance, analytical Lyapunov functions required for stability proofs are often unavailable.
Methodology
The proposed framework bridges the gap between flexible RL and rigorous stability through a two-step approach:
- Neural Lyapunov Function Learning: In the absence of analytical formulas for lossy networks, a neural network ($V_\phi$) is trained to approximate a Lyapunov function. A physics-informed loss function ensures $V_\phi$ satisfies Lyapunov criteria: it must be positive-definite and its Lie derivative (rate of change over time) must be negative, ensuring the system's energy dissipates toward a stable equilibrium.
- Lyapunov-Regularized RL: The RL controller, implemented via a Recurrent Neural Network (RNN) to capture temporal dynamics, is trained with a Lyapunov regularization term. This term penalizes actions that would cause the learned Lyapunov function to increase, effectively steering the agent away from unstable regions.
- Constrained Architecture: The controller employs a "stacked-ReLU" structure to ensure the control law is monotonic and passes through the origin, satisfying physical requirements for primary frequency control.
Experimental Results
The method was evaluated on the IEEE 39-bus (New England) system and the IEEE 300-bus system. The latter demonstrates the scalability of the approach to near-industrial levels, with training completing in approximately 15 minutes.
Key findings include:
- Stability vs. Failure: While unregularized RL agents often led the system to divergence during line disconnection simulations, the Lyapunov-regularized agent consistently stabilized the grid.
- Performance Gains: The proposed method achieved a 19% reduction in total cost (frequency deviation and control effort) compared to traditional linear droop control and a 5% improvement over standard RL.
- Robustness and Efficiency: The learned controller proved robust to topological changes. Furthermore, "warm start" training for new topologies allowed for 10x faster convergence.
Conclusion
This work demonstrates that integrating Lyapunov stability theory into RL training via neural regularization provides a powerful tool for modern power grid control. By learning a proxy for stability, we can leverage the flexibility of neural controllers without sacrificing reliability. Future work will focus on formal verification using SMT solvers and exploring the impact of communication constraints on decentralized stability.
Summary updated