Abstract
Reinforcement learning (RL) often struggles with unstable dynamical systems because standard policy gradient methods require an initial stabilizing controller to collect data. This paper introduces discount annealing, a model-free approach that bridges this gap. By solving a sequence of discounted Linear Quadratic Regulator (LQR) problems with a gradually increasing discount factor, the algorithm recovers stabilizing controllers for both linear and nonlinear systems from scratch. We provide finite-time convergence guarantees and demonstrate that the method can outperform traditional model-based LQR in terms of the region of attraction on nonlinear benchmarks.
Introduction
In control theory and reinforcement learning, the ability to stabilize a system is a prerequisite for optimization. For inherently unstable systems—such as rockets, drones, or balancing robots—the absence of a stabilizing controller leads to immediate failure, preventing the collection of useful trajectories for learning. This creates a "chicken and egg" dilemma: one needs a stable policy to learn, but learning is required to find a stable policy.
While model-based methods can linearize physics to find controllers, they require accurate system models. Model-free methods like policy gradients are more flexible but typically assume the initial policy is already stabilizing. This paper addresses this fundamental bottleneck by proposing a method to find stabilizing controllers using only a simulator.
The Stabilization Problem
Standard policy gradient guarantees often rely on the assumption that the cost function is bounded, which only holds if the system remains within a stable regime. For an unstable system, the state $x_t$ grows exponentially, causing the cost to explode.
The authors identify that the discount factor $\gamma$ in RL serves a dual purpose. Beyond weighting future rewards, it acts as a mathematical "damping" mechanism. A small $\gamma$ effectively shortens the horizon, making an unstable system appear stable over the duration of a rollout. However, a controller optimized for a very small $\gamma$ may not stabilize the true, undiscounted system. This necessitates a more sophisticated approach than simply picking a fixed discount factor.
Methodology: Discount Annealing
The proposed algorithm, Discount Annealing, leverages the stability margins of LQR controllers to transition from a damped, stable approximation to the true unstable system. The core logic relies on the fact that solving a discounted LQR problem is mathematically equivalent to solving an undiscounted problem on a "damped" version of the system.
The Algorithm
- Initialization: Start with a very small discount factor $\gamma_0$ and a zero controller ($K=0$). At this high level of damping, even the zero controller is effectively stabilizing.
- Optimization: Use policy gradients to find a near-optimal controller for the current $\gamma_t$.
- Annealing: Slightly increase the discount factor to $\gamma_{t+1}$. Because LQR controllers possess a stability margin, the controller optimized for $\gamma_t$ remains stabilizing for the slightly less damped system.
- Iteration: Repeat this process until $\gamma = 1$, which represents the true, undiscounted objective.
This approach is entirely model-free, requiring only black-box access to a simulator to provide rollouts of the system dynamics.
Theoretical and Empirical Results
Convergence Guarantees
The paper provides the first finite-time convergence guarantees for this class of problems. For linear systems (Theorem 1) and smooth nonlinear systems (Theorem 2), the algorithm is proven to find a stabilizing controller using only a polynomial number of samples. This bridges the gap between the theoretical ideal of policy gradients and the practical reality of unstable systems.
Benchmark Performance
The algorithm was evaluated using a nonlinear Cart-Pole benchmark, implemented with a 20Hz forward Euler discretization in PyTorch.
- Efficiency: The method reliably found a stabilizing policy in fewer than nine iterations of the annealing process.
- Robustness: A key finding was that the model-free controller achieved a larger Region of Attraction (ROA) ($r_{roa} \approx 0.77$) compared to the exact optimal LQR controller ($r_{roa} \approx 0.70$). This suggests that by training on the actual nonlinear dynamics rather than a linear approximation, the policy gradient method discovers a more robust control law that adapts to the system's nonlinearities.
Discussion and Limitations
While promising, the method currently requires "damped rollouts," where the simulator scales state transitions by $\sqrt{\gamma}$. This is feasible in simulation but cannot be applied directly to physical hardware where the laws of physics cannot be modified.
Furthermore, while the empirical results on the Cart-Pole are global, the theoretical guarantees for nonlinear systems are currently limited to a local neighborhood around the equilibrium. Future research may explore extending these guarantees to global nonlinear stabilization and incorporating more robust control objectives like $H_\infty$ synthesis, which explicitly accounts for worst-case disturbances.
Conclusion
Discount annealing provides a robust, model-free pathway to stabilizing unknown dynamical systems. By treating the discount factor as a tunable damping parameter, the method overcomes the initial stability requirement that has long hindered the application of RL to unstable physical systems.
Summary updated