Stabilizing Dynamical Systems via Policy Gradient Methods

Juan C. Perdomo, Jack Umenberger, Max Simchowitz

Abstract

Reinforcement learning (RL) often struggles with unstable dynamical systems because standard policy gradient methods require an initial stabilizing controller to collect data. This paper introduces discount annealing, a model-free approach that bridges this gap. By solving a sequence of discounted Linear Quadratic Regulator (LQR) problems with a gradually increasing discount factor, the algorithm recovers stabilizing controllers for both linear and nonlinear systems from scratch. We provide finite-time convergence guarantees and demonstrate that the method can outperform traditional model-based LQR in terms of the region of attraction on nonlinear benchmarks.

Introduction

In control theory and reinforcement learning, the ability to stabilize a system is a prerequisite for optimization. For inherently unstable systems—such as rockets, drones, or balancing robots—the absence of a stabilizing controller leads to immediate failure, preventing the collection of useful trajectories for learning. This creates a "chicken and egg" dilemma: one needs a stable policy to learn, but learning is required to find a stable policy.

While model-based methods can linearize physics to find controllers, they require accurate system models. Model-free methods like policy gradients are more flexible but typically assume the initial policy is already stabilizing. This paper addresses this fundamental bottleneck by proposing a method to find stabilizing controllers using only a simulator.

The Stabilization Problem

Standard policy gradient guarantees often rely on the assumption that the cost function is bounded, which only holds if the system remains within a stable regime. For an unstable system, the state $x_t$ grows exponentially, causing the cost to explode.

The authors identify that the discount factor $\gamma$ in RL serves a dual purpose. Beyond weighting future rewards, it acts as a mathematical "damping" mechanism. A small $\gamma$ effectively shortens the horizon, making an unstable system appear stable over the duration of a rollout. However, a controller optimized for a very small $\gamma$ may not stabilize the true, undiscounted system. This necessitates a more sophisticated approach than simply picking a fixed discount factor.

Methodology: Discount Annealing

The proposed algorithm, Discount Annealing, leverages the stability margins of LQR controllers to transition from a damped, stable approximation to the true unstable system. The core logic relies on the fact that solving a discounted LQR problem is mathematically equivalent to solving an undiscounted problem on a "damped" version of the system.

The Algorithm

  1. Initialization: Start with a very small discount factor $\gamma_0$ and a zero controller ($K=0$). At this high level of damping, even the zero controller is effectively stabilizing.
  2. Optimization: Use policy gradients to find a near-optimal controller for the current $\gamma_t$.
  3. Annealing: Slightly increase the discount factor to $\gamma_{t+1}$. Because LQR controllers possess a stability margin, the controller optimized for $\gamma_t$ remains stabilizing for the slightly less damped system.
  4. Iteration: Repeat this process until $\gamma = 1$, which represents the true, undiscounted objective.

This approach is entirely model-free, requiring only black-box access to a simulator to provide rollouts of the system dynamics.

Theoretical and Empirical Results

Convergence Guarantees

The paper provides the first finite-time convergence guarantees for this class of problems. For linear systems (Theorem 1) and smooth nonlinear systems (Theorem 2), the algorithm is proven to find a stabilizing controller using only a polynomial number of samples. This bridges the gap between the theoretical ideal of policy gradients and the practical reality of unstable systems.

Benchmark Performance

The algorithm was evaluated using a nonlinear Cart-Pole benchmark, implemented with a 20Hz forward Euler discretization in PyTorch.

  • Efficiency: The method reliably found a stabilizing policy in fewer than nine iterations of the annealing process.
  • Robustness: A key finding was that the model-free controller achieved a larger Region of Attraction (ROA) ($r_{roa} \approx 0.77$) compared to the exact optimal LQR controller ($r_{roa} \approx 0.70$). This suggests that by training on the actual nonlinear dynamics rather than a linear approximation, the policy gradient method discovers a more robust control law that adapts to the system's nonlinearities.

Discussion and Limitations

While promising, the method currently requires "damped rollouts," where the simulator scales state transitions by $\sqrt{\gamma}$. This is feasible in simulation but cannot be applied directly to physical hardware where the laws of physics cannot be modified.

Furthermore, while the empirical results on the Cart-Pole are global, the theoretical guarantees for nonlinear systems are currently limited to a local neighborhood around the equilibrium. Future research may explore extending these guarantees to global nonlinear stabilization and incorporating more robust control objectives like $H_\infty$ synthesis, which explicitly accounts for worst-case disturbances.

Conclusion

Discount annealing provides a robust, model-free pathway to stabilizing unknown dynamical systems. By treating the discount factor as a tunable damping parameter, the method overcomes the initial stability requirement that has long hindered the application of RL to unstable physical systems.

Summary updated

BibTeX
@inproceedings{perdomo2021stabilizing,
  author = {Juan C. Perdomo and Jack Umenberger and Max Simchowitz},
  title = {{Stabilizing Dynamical Systems via Policy Gradient Methods}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=yxg-i8DAHK},
}
Figure