TL;DR
This paper introduces a method for "robust reinforcement learning" in linear systems that balances exploration and exploitation by solving a convex optimization problem. Instead of exploring randomly, the agent specifically excites the system to reduce uncertainty in the parameters that most impact the worst-case cost. This ensures stable and efficient learning even in the face of significant model uncertainty.
The Dual Control Problem in LQR
In Reinforcement Learning (RL), agents constantly face the "exploration-exploitation" tradeoff: should they stick with the best policy found so far (exploit) or try new actions to learn more about the environment (explore)?
In the context of control theory—specifically Linear Quadratic Regulators (LQR)—most existing methods either explore randomly or assume an "optimistic" view of the world. However, in real-world engineering applications like robotics, arbitrary exploration can be dangerous or highly inefficient. The authors tackle the "dual control problem": how to design a control policy that simultaneously achieves a goal (minimizing cost) and gathers the right information to improve future performance, all while guaranteeing robustness against current model uncertainty.
Robust Reinforcement Learning Framework
The authors propose a "Robust Reinforcement Learning" (RRL) framework built on four key technical pillars:
- Spectral Uncertainty Bounds: Instead of using standard ellipsoidal regions to describe model uncertainty, the authors derive a high-probability bound on the spectral norm of the estimation error. This specific mathematical form is crucial because it allows the subsequent optimization problem to be formulated as a convex program.
- Convex Approximation of Worst-Case Cost: Calculating the exact worst-case cost for an uncertain system is typically a non-convex (and thus computationally "hard") problem. The authors use Semidefinite Programming (SDP) to create a convex approximation that upper-bounds the infinite-horizon LQR cost.
- Targeted Exploration via SDP: The control policy is defined as $u_t = Kx_t + \Sigma^{1/2}e_t$, where $K$ is the feedback gain and $\Sigma$ is the exploration noise covariance. By solving an SDP, the method optimizes $K$ and $\Sigma$ simultaneously. This ensures the exploration noise $\Sigma$ isn't just "white noise" but is shaped to excite the system in ways that reduce uncertainty in the parameters the controller is most sensitive to.
- Receding Horizon Application: The agent periodically updates its model and re-solves the optimization problem as new data arrives. This allows the policy to naturally evolve from "cautious and exploratory" to "aggressive and exploitative" as uncertainty decreases.
Experimental Setup
The paper evaluates the method using two primary setups:
- Numerical Simulation: A synthetic 3-state, 2-input linear dynamical system designed to test the algorithm's ability to handle unstable open-loop dynamics.
- Hardware-in-the-Loop (HIL): An experiment involving a physical Quanser QUBE 2 servo-mechanism (a rotary pendulum system) interfaced with a simulated dynamical system. This provides a bridge between pure theory and real-world physical constraints like friction and sampling rates.
Key Insights
- Exploration is a Means, Not an End: The authors argue that reducing uncertainty has no intrinsic value unless it helps achieve the primary goal. Therefore, exploration should be "targeted"—only exciting the system in directions that actually matter for the specific cost function being minimized.
- Robustness vs. Optimism: Unlike "Optimism in the Face of Uncertainty" (OFU) methods which assume the best-case scenario, this method optimizes for the worst-case. This makes it significantly safer for physical systems where an optimistic guess might lead to instability.
- Automated Tradeoff: The balance between exploration and exploitation isn't a manual "tuning knob"; it emerges automatically from the solution to the convex optimization problem.
Key Results
- Stability: In the hardware-in-the-loop experiment, popular methods like Thompson Sampling failed to stabilize the system in several trials, leading to divergent behavior. The proposed RRL method remained stable across all trials.
- Performance: The RRL method achieved lower total costs compared to "greedy" robust methods. While greedy methods had lower costs initially, the RRL method's targeted exploration led to a much faster reduction in uncertainty, allowing it to significantly outperform the greedy baseline in later stages.
- Information Efficiency: Interestingly, the "greedy exploration" baseline (which maximizes information gain) actually gathered more total information than RRL, but RRL achieved a lower cost. This confirms that RRL successfully ignores "useless" information and focuses only on what is necessary for control.
Limitations and Future Work
- Linearity Assumption: The method is strictly designed for Linear Time-Invariant (LTI) systems. Extending this to non-linear systems would likely break the convex guarantees provided by the SDP formulation.
- Computational Complexity: Solving Semidefinite Programs can be computationally expensive as the number of states ($n_x$) and inputs ($n_u$) grows. The authors note the complexity scales roughly with $O(m^3)$ where $m$ is related to the number of decision variables, which may limit its use in very high-dimensional systems without further optimization.
- Known Noise Statistics: The method assumes the process noise covariance ($\sigma_w^2$) is known or can be accurately estimated, which may not always be the case in noisy industrial environments.
Summary updated