Abstract
This paper investigates the inherent robustness of policy iteration for Linear Quadratic Regulation (LQR) against learning errors. By framing the learning process as a discrete-time dynamical system, the authors utilize control-theoretic tools like Input-to-State Stability (ISS) to prove that small errors in policy evaluation do not lead to catastrophic instability. Instead, the algorithm converges to a bounded neighborhood of the optimal solution. The study introduces Optimistic Least-Squares Policy Iteration (O-LSPI), a model-free algorithm that demonstrates superior stability and convergence properties in numerical benchmarks.
Introduction
Reinforcement Learning (RL) in continuous state and action spaces often faces challenges due to noise from function approximation, sensor inaccuracies, and finite data. In safety-critical applications like robotics or autonomous driving, which are frequently modeled as Linear Quadratic Regulators (LQR), a single erroneous policy update can lead to system instability. While policy iteration has been used for decades, a rigorous mathematical guarantee of its robustness to such learning errors was previously missing. This research addresses the fundamental question: under what conditions is policy iteration robustly stable?
Methodology
The core innovation of this work is viewing the policy iteration algorithm itself as a dynamical system.
Stability Analysis
The authors apply concepts of Exponential Stability and Input-to-State Stability (ISS) to the matrix-space trajectories of the algorithm. They prove that the optimal LQR solution is a locally exponentially stable equilibrium for exact policy iteration. Furthermore, when learning errors are treated as disturbance inputs, the algorithm is ISS. This ensures that the distance between the learned policy and the optimal one remains bounded by the magnitude of the learning error.
Optimistic Least-Squares Policy Iteration (O-LSPI)
Building on this theoretical foundation, the authors propose O-LSPI. This off-policy, model-free method is designed for systems with additive stochastic noise. A key advantage of O-LSPI is its efficiency; it requires solving a pseudo-inverse problem only once and does not necessitate knowledge of the system's noise characteristics (the $C$ matrix), making it highly practical for real-world applications.
Results
The proposed framework was validated using a standard 3-dimensional state, 2-dimensional control LQR benchmark. The evaluation focused on the "fraction of stability" (the percentage of trials resulting in a stable controller) and the relative error of the cost function.
Key findings include:
- Robustness (Theorem 2): Theoretical proof that for any initial stabilizing control gain, bounded learning errors lead to a solution that remains within a small neighborhood of the optimum.
- Convergence (Theorem 3): O-LSPI converges almost surely to the optimal solution as data ($M$) and iterations ($T$) approach infinity.
- Numerical Performance: As shown in Figure 1, as the number of data samples ($M$) increases, both the average relative error and its variance decrease significantly. For $M > 10^4$, the fraction of stable trials reaches 100%. O-LSPI demonstrates comparable or superior performance to existing methods like LSPIv1, particularly in its ability to maintain stability with fewer prior assumptions about system noise.
Discussion and Conclusion
The study reveals that policy iteration for LQR is inherently robust due to its mathematical structure, requiring no additional "robustness" layers to handle small-scale noise. By quantifying the "neighborhood" of the optimal solution based on error bounds, practitioners can predict the performance limits of their learned controllers.
While these results provide a strong foundation for linear systems, extending this dynamical system perspective to nonlinear RL remains a primary area for future work. The success of this approach suggests that viewing learning algorithms through the lens of control theory can provide deep insights into the stability and convergence of modern AI systems.
Summary updated