Robust exploration in linear quadratic reinforcement learning

Jack Umenberger, Mina Ferizbegovic, Thomas B. Schön, Håkan Hjalmarsson

Abstract

This paper concerns the problem of learning control policies for an unknown linear dynamical system to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task robustly: i.e., we minimize the worst-case cost, accounting for system uncertainty given the observed data. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both.

Introduction: The Safety Challenge in RL

While Reinforcement Learning (RL) has achieved remarkable success in virtual environments, its application to physical systems—such as chemical plants or robotics—remains challenging. In these real-world settings, random exploration can lead to catastrophic failures. The core difficulty lies in the "dual control problem": the trade-off between exploitation (using the current model to minimize cost) and exploration (perturbing the system to improve the model).

Traditional RL often focuses on minimizing "regret," which can lead to aggressive exploration that ignores stability. In contrast, engineering applications demand robustness—a guarantee that the controller will remain stable even if the model is imperfect. This paper introduces a mathematically rigorous framework for "robust reinforcement learning" in linear systems that balances these competing needs.

Methodology: Targeted Exploration via Convex Optimization

The authors address the Linear Quadratic Regulator (LQR) problem with unknown dynamics ($A$ and $B$ matrices). Their approach is built on three pillars:

  1. Quantifying Uncertainty: Instead of a point estimate, the method defines a "credibility region"—a set of all models consistent with observed data, bounded by a high-probability spectral norm.
  2. Robust Policy Synthesis: The control problem is formulated as a Semidefinite Program (SDP). By optimizing for the worst-case model within the credibility region, the resulting controller ensures stability and performance under uncertainty.
  3. Targeted Exploration: The optimization determines both a safe gain ($K$) and an optimal excitation signal ($\Sigma$). This ensures the system is excited only in ways that reduce uncertainty for parameters most critical to performance.
Conceptual illustration of the receding horizon strategy, balancing cost reduction with information gain over successive epochs.

The process follows a receding horizon strategy: the optimization is solved, the policy is applied for a short "epoch," and the uncertainty bounds are updated with new data before the next cycle.

Experimental Results

The Robust Reinforcement Learning (RRL) method was evaluated through numerical simulations of unstable systems and hardware-in-the-loop (HIL) experiments using a Quanser QUBE-Servo 2 rotary pendulum.

Performance comparison showing the evolution of cost and uncertainty across training epochs for different exploration strategies.

Key findings include:

  • Superior Performance: RRL consistently achieved lower cumulative costs than both "Nominal" (greedy) and "Greedy Exploration" approaches.
  • Information Efficiency: RRL gathered more relevant information than the nominal method while being more efficient than pure exploration, specifically targeting parameters vital for control.
  • Robustness and Stability: In physical hardware tests, RRL demonstrated significantly tighter performance bounds, indicating higher reliability and less variance in cost compared to alternatives.

Conclusion and Future Directions

The proposed framework demonstrates that robustness can serve as a natural regulator for exploration. By optimizing for the worst case, the algorithm is inherently cautious when data is sparse and becomes more aggressive as uncertainty decreases.

While currently limited to linear time-invariant (LTI) systems and subject to some conservatism due to convex approximations, future work aims to extend this to non-linear dynamics and systems with explicit state and input constraints (e.g., safety limits on motor temperature or actuator range).

Summary updated

Abstract

This paper concerns the problem of learning control policies for an unknown linear dynamical system to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task robustly: i.e., we minimize the worst-case cost, accounting for system uncertainty given the observed data. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both.
BibTeX
@inproceedings{umenberger2019robust,
  author = {Jack Umenberger and Mina Ferizbegovic and Thomas B. Schön and Håkan Hjalmarsson},
  title = {{Robust exploration in linear quadratic reinforcement learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=gt4yrL4p5d},
}
Figure