Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

Xin Guo, Xinyu Li, Renyuan Xu

TL;DR

This paper introduces two new algorithms, Regularized Policy Gradient (RPG) and Iterative Policy Optimization (IPO), designed to solve the classic Linear-Quadratic Control (LQC) problem using entropy regularization. The standout result is that IPO achieves super-linear local convergence (order 1.5), significantly outperforming standard methods and enabling highly efficient transfer learning between similar environments.

What Problem Does This Paper Address?

Linear-Quadratic Control (LQC) is the "Hello World" of control theory—a foundational model where a system moves linearly and we want to minimize a quadratic cost. While well-understood in classical control, LQC remains a vital benchmark for Reinforcement Learning (RL).

Most traditional LQC approaches focus on deterministic policies. However, modern RL often uses entropy regularization to encourage exploration and prevent the agent from collapsing onto a single action too quickly. While entropy regularization is known to "smooth" the optimization landscape, we lacked rigorous, non-asymptotic convergence rates for these stochastic policies in the LQC setting.

The authors tackle a critical challenge: How can we design algorithms that aren't just "guaranteed to eventually work," but are provably fast and sample-efficient enough for high-stakes applications like finance or healthcare?

Methods

The authors propose two primary algorithms for infinite-horizon, discounted LQC problems:

  1. Regularized Policy Gradient (RPG): A first-order method that updates the policy mean ($K$) and covariance ($\Sigma$) simultaneously. It incorporates the discounted state correlation matrix as a preconditioner, which helps account for the geometry of the parameter space.
  2. Iterative Policy Optimization (IPO): A more aggressive approach that solves a one-step optimization problem at each iteration. It is conceptually similar to the Gauss-Newton method but adapted for entropy-regularized Gaussian policies.

Unlike standard LQC, which results in a single optimal action, these methods learn a Gaussian policy. This means the agent learns both the best average action and the optimal amount of "noise" (exploration) to maintain.

Datasets

As a theoretical control paper, the authors do not use external real-world datasets. Instead, they evaluate their algorithms using synthetic numerical simulations.

  • Setup: System matrices ($A, B$) and cost matrices ($Q, R$) are generated randomly.
  • Scale: Experiments were conducted with state dimensions up to $n=400$ and control dimensions up to $k=200$.
  • Benchmarks: The methods were compared against standard Gauss-Newton updates with fixed covariance to demonstrate the benefit of dynamically updating the policy's exploration noise.

Key Insights and Ideas

  • The "Smoothing" Effect: Entropy regularization doesn't just help with exploration; it establishes a "gradient dominance" condition. This ensures that as long as the gradient is non-zero, there is a clear path to the global optimum, preventing the algorithm from getting stuck.
  • Super-linear Speedup: The most significant "aha moment" is that IPO achieves a convergence rate of order 1.5 once it gets close to the optimum. In practical terms, this means the error drops precipitously (e.g., from $10^{-2}$ to $10^{-14}$) in just a few iterations.
  • Transfer Learning as a Head Start: The authors prove that if you have an optimal policy for a "well-understood" environment, you can use it as a starting point for a "new" similar environment. If the environments are close enough, the IPO algorithm starts immediately in its super-linear convergence regime, learning the new task almost instantly.

Key Results

  • Global Linear Convergence: Both RPG and IPO are proven to converge linearly from any valid initialization.
  • Local Super-linear Convergence: IPO is shown to have a convergence rate of order 1.5. In numerical tests, IPO reached a normalized error of $10^{-14}$ within just 6 iterations.
  • Transfer Efficiency: In transfer learning scenarios, IPO reduced the error to below $10^{-12}$ in only two epochs when initialized with a policy from a perturbed but similar environment.
  • Covariance Matters: The experiments highlight that updating the covariance matrix ($\Sigma$) dynamically is crucial. Methods that keep exploration noise fixed converge much slower than the proposed IPO.

Limitations and Future Work

  • Exact Policy Evaluation: The current theoretical proofs assume "exact policy evaluation," meaning the agent knows the exact value of its current policy. In real-world RL, this must be estimated from samples, which introduces noise.
  • Model-Free Extension: While the paper discusses a model-free version (Algorithm 7.1) using zeroth-order gradient estimates, the rigorous convergence rates for the finite-sample case are left for future work.
  • Linearity Assumption: The results are specific to Linear-Quadratic systems. Extending these super-linear guarantees to non-linear systems or more complex neural network policies remains a significant open challenge.

Summary updated

Abstract

This paper proposes and analyzes two new policy learning methods: regularized policy gradient (RPG) and iterative policy optimization (IPO), for a class of discounted linear-quadratic control (LQC) problems over an infinite time horizon with entropy regularization. Assuming access to the exact policy evaluation, both proposed approaches are proven to converge linearly in finding optimal policies of the regularized LQC. Moreover, the IPO method can achieve a super-linear convergence rate once it enters a local region around the optimal policy. Finally, when the optimal policy for an RL problem with a known environment is appropriately transferred as the initial policy to an RL problem with an unknown environment, the IPO method is shown to enable a super-linear convergence rate if the two environments are sufficiently close. Performances of these proposed algorithms are supported by numerical examples.
BibTeX
@misc{guo2023fast,
  author = {Xin Guo and Xinyu Li and Renyuan Xu},
  title = {{Fast Policy Learning for Linear Quadratic Control with Entropy Regularization}},
  year = {2023},
  eprint = {2311.14168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.14168},
}
Figure