Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems

Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, Anima Anandkumar

Introduction: The Stabilization Catch-22

In control theory and reinforcement learning (RL), the Linear Quadratic Regulator (LQR) is a fundamental framework for managing systems with linear dynamics and quadratic costs. However, a significant challenge arises when the system's underlying physics are unknown. A learning agent faces a "catch-22": to learn the system, it must interact with it, but without a stabilizing controller, the system's state might "blow up" (diverge to infinity) before sufficient data is collected.

Existing algorithms often suffer from exponential regret, where the error grows uncontrollably as the system's complexity (number of states $n$ and inputs $d$) increases. Other methods bypass this by assuming the agent starts with a pre-existing stabilizing controller. This work introduces StabL, an algorithm designed to stabilize and optimize unknown linear systems from scratch, even in the broad and realistic category of stabilizable systems.

The StabL Algorithm

The authors propose StabL (Stabilizing Learning), which operates in two distinct phases to ensure both safety and efficiency:

  1. Early Improved Exploration: During an initial period ($T_w$), the agent employs an optimistic controller augmented with isotropic Gaussian noise. These random perturbations force the system to reveal its dynamics across all dimensions, ensuring "persistence of excitation" and preventing the agent from ignoring potentially critical but "expensive" parts of the state space.
  2. Stabilizing Adaptive Control: Once the system is likely stabilized, the agent transitions to the Optimism in the Face of Uncertainty (OFU) principle. It maintains a confidence set of possible models and selects the one yielding the lowest cost, refining its estimates over time.

To maintain stability during learning, StabL uses a policy update rule based on "determinant doubling." The controller is only updated when the agent has acquired significantly more information (measured by the determinant of the design matrix) or after a specific time horizon, preventing jittery changes that could destabilize the system.

Theoretical Advancements: From Exponential to Polynomial

A key contribution of this work is the reduction of regret's dependence on system dimensions. By ensuring rapid stabilization, the authors prove that StabL's regret depends only polynomially on $n$ and $d$. This is a massive leap over previous "from scratch" methods that featured catastrophic $(n+d)^{n+d}$ terms.

Table 1: Comparison of StabL with prior work in terms of regret, system settings, and controller requirements.

Furthermore, while most prior work assumes "controllability" (the ability to reach any state), StabL operates under the more practical assumption of "stabilizability" (the ability to return to equilibrium), making it more applicable to real-world engineering tasks.

Experimental Results

The paper evaluates StabL across several simulated tasks, including a marginally unstable Laplacian system, a Boeing 747 flight control model, and a UAV flight model.

Table 2: Empirical stability results comparing the maximum state norm of StabL against other popular methods.

In these experiments, StabL demonstrated superior stability. For instance, in the Boeing 747 and Laplacian tasks, the "state norm"—a measure of system deviation—remained orders of magnitude smaller than standard OFU methods. In the Laplacian system, StabL's average regret was approximately six orders of magnitude lower than the classic OFULQ algorithm after 200 steps, effectively preventing the "blow-ups" that plagued other learners.

Conclusion and Future Work

StabL achieves the theoretical "gold standard" of $\tilde{\mathcal{O}}(\sqrt{T})$ regret while providing robust stabilization for unknown systems. While highly effective, future research could extend this framework to partially observable systems (the LQG problem) and address the challenges of non-linear dynamics and the computational complexity of solving optimistic models in high-dimensional spaces.

Summary updated

Abstract

In this work, we study model-based reinforcement learning (RL) in unknown stabilizable linear dynamical systems. When learning a dynamical system, one needs to stabilize the unknown dynamics in order to avoid system blow-ups. We propose an algorithm that certifies fast stabilization of the underlying system by effectively exploring the environment with an improved exploration strategy. We show that the proposed algorithm attains $\tilde{\mathcal{O}}(\sqrt{T})$ regret after $T$ time steps of agent-environment interaction. We also show that the regret of the proposed algorithm has only a polynomial dependence in the problem dimensions, which gives an exponential improvement over the prior methods. Our improved exploration method is simple, yet efficient, and it combines a sophisticated exploration policy in RL with an isotropic exploration strategy to achieve fast stabilization and improved regret. We empirically demonstrate that the proposed algorithm outperforms other popular methods in several adaptive control tasks.
BibTeX
@misc{lale2020reinforcement,
  author = {Sahin Lale and Kamyar Azizzadenesheli and Babak Hassibi and Anima Anandkumar},
  title = {{Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems}},
  year = {2020},
  eprint = {2007.12291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.12291},
}
Figure