Abstract
Ensuring the stability of Reinforcement Learning (RL) policies is a critical challenge for deploying learned controllers in safety-critical systems. Classical Lyapunov-based stability certificates often fail for RL policies due to strict stepwise decrease requirements and the non-monotonic nature of complex tasks. We present a framework using Generalized Lyapunov Functions (GLFs) that relaxes these constraints by employing a multi-step weighted average condition. By augmenting RL value functions with learned neural residuals and utilizing formal verification tools like $\alpha, \beta$-CROWN, we demonstrate significantly improved stability certification for nonlinear systems and tighter bounds for linear systems.
Introduction
In classical control theory, stability is traditionally guaranteed by finding a Lyapunov function—a mathematical "energy" function that must strictly decrease at every time step. However, modern Reinforcement Learning (RL) policies, typically represented by deep neural networks, rarely satisfy this strict condition. RL value functions, while natural candidates for Lyapunov functions, frequently exhibit non-monotonic behavior, especially in tasks requiring temporary energy increases to build momentum, such as a pendulum swing-up. This paper addresses these limitations by introducing a multi-step stability criterion that allows for short-term fluctuations while ensuring long-term convergence.
Methodology
The proposed framework introduces several key innovations to bridge the gap between RL and control theory:
- Generalized Lyapunov Condition: Instead of requiring a one-step decrease ($V(x_{k+1}) < V(x_k)$), stability is certified if a weighted average of function values over a horizon of $M$ steps is less than the current value. This allows for temporary "non-monotonic" increases in energy, provided the system settles over the horizon.
- Value Function Augmentation: The certificate is constructed by taking a pre-trained RL value function and augmenting it with a learned neural residual term. This leverages the existing task topology encoded in the value function while allowing the residual to correct local stability violations.
- Joint Synthesis and Verification: For new tasks, a multi-step Lyapunov loss is used to jointly train a stable neural controller and its corresponding certificate. Formal verification is performed using the $\alpha, \beta$-CROWN verifier to provide rigorous proofs across the entire state space.
Experimental Results
The method was evaluated on standard continuous control benchmarks, including Inverted Pendulum, Cartpole, Path Tracking, and a 2D Quadrotor.
Improved Stability Bounds and LQR Efficiency
For linear systems (LQR), the generalized approach provides much less conservative stability bounds. In a standard LQR example, the method certified stability for a discount factor of $\gamma > 0.6229$, significantly outperforming the classical baseline of $\gamma > 0.8090$. As shown in Figure 1, increasing the horizon $M$ is key to reaching these tighter bounds, eventually approaching the theoretical limit of $1/3$.
Increased Certified Regions
In the Inverted Pendulum task, moving from a 1-step ($M=1$) to a 3-step ($M=3$) certificate doubled the volume of the Certified Region of Attraction (ROA) from 42.9 to 89.2. The framework successfully certified 100% of sampled test states for policies trained via PPO, SAC, and TD-MPC on Pendulum and Cartpole benchmarks, where traditional one-step methods failed.
Discussion and Conclusion
The use of multi-step horizons is essential for certifying tasks like "swing-up" where energy must transiently increase to gain momentum. While the approach scales to 6D systems like the 2D Quadrotor, challenges remain in automatically selecting the optimal horizon $M$ and managing the computational cost of formal verification for higher-dimensional systems. Future work will explore unknown equilibrium states and further optimization of the verification process to handle high-dimensional robotics.
Summary updated