Robust exploration in linear quadratic reinforcement learning

Jack Umenberger, Mina Ferizbegovic, Thomas B. Schön, Håkan Hjalmarsson

TL;DR

This paper introduces a method for "robust reinforcement learning" in linear systems that balances exploration and exploitation by solving a convex optimization problem. Instead of exploring randomly, the agent specifically excites the system to reduce uncertainty in the parameters that most impact the worst-case cost. This ensures stable and efficient learning even in the face of significant model uncertainty.

The Dual Control Problem in LQR

In Reinforcement Learning (RL), agents constantly face the "exploration-exploitation" tradeoff: should they stick with the best policy found so far (exploit) or try new actions to learn more about the environment (explore)?

In the context of control theory—specifically Linear Quadratic Regulators (LQR)—most existing methods either explore randomly or assume an "optimistic" view of the world. However, in real-world engineering applications like robotics, arbitrary exploration can be dangerous or highly inefficient. The authors tackle the "dual control problem": how to design a control policy that simultaneously achieves a goal (minimizing cost) and gathers the right information to improve future performance, all while guaranteeing robustness against current model uncertainty.

Cartoon depiction of the receding horizon policy application, illustrating the tradeoff between minimizing worst-case cost and gathering information over time.

Robust Reinforcement Learning Framework

The authors propose a "Robust Reinforcement Learning" (RRL) framework built on four key technical pillars:

  1. Spectral Uncertainty Bounds: Instead of using standard ellipsoidal regions to describe model uncertainty, the authors derive a high-probability bound on the spectral norm of the estimation error. This specific mathematical form is crucial because it allows the subsequent optimization problem to be formulated as a convex program.
  2. Convex Approximation of Worst-Case Cost: Calculating the exact worst-case cost for an uncertain system is typically a non-convex (and thus computationally "hard") problem. The authors use Semidefinite Programming (SDP) to create a convex approximation that upper-bounds the infinite-horizon LQR cost.
  3. Targeted Exploration via SDP: The control policy is defined as $u_t = Kx_t + \Sigma^{1/2}e_t$, where $K$ is the feedback gain and $\Sigma$ is the exploration noise covariance. By solving an SDP, the method optimizes $K$ and $\Sigma$ simultaneously. This ensures the exploration noise $\Sigma$ isn't just "white noise" but is shaped to excite the system in ways that reduce uncertainty in the parameters the controller is most sensitive to.
  4. Receding Horizon Application: The agent periodically updates its model and re-solves the optimization problem as new data arrives. This allows the policy to naturally evolve from "cautious and exploratory" to "aggressive and exploitative" as uncertainty decreases.

Experimental Setup

The paper evaluates the method using two primary setups:

  • Numerical Simulation: A synthetic 3-state, 2-input linear dynamical system designed to test the algorithm's ability to handle unstable open-loop dynamics.
  • Hardware-in-the-Loop (HIL): An experiment involving a physical Quanser QUBE 2 servo-mechanism (a rotary pendulum system) interfaced with a simulated dynamical system. This provides a bridge between pure theory and real-world physical constraints like friction and sampling rates.

Key Insights

  • Exploration is a Means, Not an End: The authors argue that reducing uncertainty has no intrinsic value unless it helps achieve the primary goal. Therefore, exploration should be "targeted"—only exciting the system in directions that actually matter for the specific cost function being minimized.
  • Robustness vs. Optimism: Unlike "Optimism in the Face of Uncertainty" (OFU) methods which assume the best-case scenario, this method optimizes for the worst-case. This makes it significantly safer for physical systems where an optimistic guess might lead to instability.
  • Automated Tradeoff: The balance between exploration and exploitation isn't a manual "tuning knob"; it emerges automatically from the solution to the convex optimization problem.

Key Results

Performance comparison showing cost and information over time, alongside total cost distributions. The RRL method achieves lower total costs and better stability compared to baselines.
  • Stability: In the hardware-in-the-loop experiment, popular methods like Thompson Sampling failed to stabilize the system in several trials, leading to divergent behavior. The proposed RRL method remained stable across all trials.
  • Performance: The RRL method achieved lower total costs compared to "greedy" robust methods. While greedy methods had lower costs initially, the RRL method's targeted exploration led to a much faster reduction in uncertainty, allowing it to significantly outperform the greedy baseline in later stages.
  • Information Efficiency: Interestingly, the "greedy exploration" baseline (which maximizes information gain) actually gathered more total information than RRL, but RRL achieved a lower cost. This confirms that RRL successfully ignores "useless" information and focuses only on what is necessary for control.

Limitations and Future Work

  • Linearity Assumption: The method is strictly designed for Linear Time-Invariant (LTI) systems. Extending this to non-linear systems would likely break the convex guarantees provided by the SDP formulation.
  • Computational Complexity: Solving Semidefinite Programs can be computationally expensive as the number of states ($n_x$) and inputs ($n_u$) grows. The authors note the complexity scales roughly with $O(m^3)$ where $m$ is related to the number of decision variables, which may limit its use in very high-dimensional systems without further optimization.
  • Known Noise Statistics: The method assumes the process noise covariance ($\sigma_w^2$) is known or can be accurately estimated, which may not always be the case in noisy industrial environments.

Summary updated

Abstract

Learning to make decisions in an uncertain and dynamic environment is a task of fundamental performance in a number of domains. This paper concerns the problem of learning control policies for an unknown linear dynamical system so as to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task ‘robustly’, i.e., the worst-case cost, accounting for system uncertainty given the observed data, is minimized. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism are used to demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both.
BibTeX
@inproceedings{umenberger2019robust,
  author = {Jack Umenberger and Mina Ferizbegovic and Thomas B. Schön and Håkan Hjalmarsson},
  title = {{Robust exploration in linear quadratic reinforcement learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=gt4yrL4p5d},
}
Figure