Abstract
In this work, we study model-based reinforcement learning (RL) in unknown stabilizable linear dynamical systems. When learning a dynamical system, one needs to stabilize the unknown dynamics in order to avoid system blow-ups. We propose an algorithm that certifies fast stabilization of the underlying system by effectively exploring the environment with an improved exploration strategy. We show that the proposed algorithm attains $\tilde{\mathcal{O}}(\sqrt{T})$ regret after $T$ time steps of agent-environment interaction. We also show that the regret of the proposed algorithm has only a polynomial dependence in the problem dimensions, which gives an exponential improvement over the prior methods. Our improved exploration method is simple, yet efficient, and it combines a sophisticated exploration policy in RL with an isotropic exploration strategy to achieve fast stabilization and improved regret. We empirically demonstrate that the proposed algorithm outperforms other popular methods in several adaptive control tasks.
Introduction
Reinforcement learning (RL) in unknown linear dynamical systems presents a fundamental challenge: balancing exploration with system stability. In Linear Quadratic Regulator (LQR) problems, an agent must learn to control a system with unknown dynamics while minimizing a quadratic cost. A critical risk is "system blow-up," where poor control leads to state values diverging to infinity before sufficient learning occurs. Traditionally, this has been mitigated by requiring a "warm start" with a pre-existing stabilizing controller. However, achieving efficient learning from scratch—without such a prior—remains a significant hurdle.
The Challenge of Stabilization and Regret
Previous state-of-the-art methods, such as "Optimism in the Face of Uncertainty" (OFU), often struggle in the absence of an initial stabilizing controller. Their regret—the difference between the accumulated cost and that of an optimal controller—can scale exponentially with the system dimensions ($n$ states and $d$ inputs). This exponential dependence makes them impractical for high-dimensional systems. Furthermore, purely optimistic agents may avoid "expensive" states that are actually necessary for identifying the system's stabilizing parameters, leading to delayed stabilization and potential catastrophic failure.
Methodology: The StabL Algorithm
The authors introduce StabL (Stabilizing Learning), an algorithm designed to certify fast stabilization and achieve order-optimal regret with polynomial dependence on system dimensions. StabL's architecture is built on two key innovations:
- Isotropic Exploration for Fast Identification: Unlike standard RL agents that prioritize cost minimization from the start, StabL employs an initial phase of isotropic Gaussian noise injection. This strategy ensures that all system dimensions are sufficiently excited, allowing the agent to rapidly identify the underlying dynamics and find a stabilizing controller.
- Safe Policy Switching: Frequent updates to a controller can destabilize even a known system. StabL utilizes a "determinant doubling" rule and a minimum time commitment ($H_0$) for each policy. This ensures that the switching frequency is controlled, preventing state explosion during the learning process.
As shown in the table above, StabL is the first to achieve $\tilde{\mathcal{O}}(\sqrt{T})$ regret in the general "stabilizable" setting without requiring a prior stabilizing controller, while maintaining polynomial dependence on system dimensions. This represents an exponential improvement over previous methods like Abbasi-Yadkori and Szepesvári (2011).
Empirical Performance
The effectiveness of StabL was evaluated across several challenging benchmarks, including a Boeing 747 flight control model and a marginally unstable Laplacian system. A key metric for success in these tasks is the maximum state norm ($\Vert x \Vert_2$), which indicates how well the algorithm prevents system blow-ups.
In empirical tests, StabL consistently outperformed popular baselines like OFULQ and Certainty Equivalence Control (CEC). While other methods often saw state values explode to $10^{10}$ or higher, StabL kept them tightly bounded (around $10^1$), demonstrating its robustness in adaptive control tasks.
Conclusion
StabL represents a significant step forward in model-based RL for linear systems. By prioritizing early, isotropic exploration, it "certifies" stability and reduces the dimensional dependence of regret from exponential to polynomial. Future work includes extending these guarantees to partially observable systems (LQG) and exploring how these stabilization principles might apply to more complex, non-linear dynamics.
Summary updated