Data-Enabled Policy Optimization for the Linear Quadratic Regulator

Feiran Zhao, Florian Dörfler, Keyou You

Abstract

Policy optimization (PO), an essential approach of reinforcement learning for a broad range of system classes, requires significantly more system data than indirect (identification-followed-by-control) methods or behavioral-based direct methods even in the simplest linear quadratic regulator (LQR) problem. In this paper, we take an initial step towards bridging this gap by proposing the data-enabled policy optimization (DeePO) method, which requires only a finite number of sufficiently exciting data to iteratively solve the LQR problem via PO. Based on a data-driven closed-loop parameterization, we are able to directly compute the policy gradient from a batch of persistently exciting data. Next, we show that the nonconvex PO problem satisfies a projected gradient dominance property by relating it to an equivalent convex program, leading to the global convergence of DeePO. Moreover, we apply regularization methods to enhance certainty-equivalence and robustness of the resulting controller and show an implicit regularization property. Finally, we perform simulations to validate our results.

Introduction

In the field of data-driven control, a significant "sample complexity gap" exists between indirect methods—which identify a model before designing a controller—and Policy Optimization (PO), the cornerstone of modern Reinforcement Learning (RL). While indirect methods are data-efficient, they are often complex to implement. Conversely, PO is conceptually simple but notoriously data-hungry, typically requiring thousands of system trajectories to estimate gradients via zeroth-order optimization.

This research introduces DeePO (Data-enabled Policy Optimization), a framework designed to make RL as data-efficient as classical control for the fundamental Linear Quadratic Regulator (LQR) problem. By computing policy gradients directly from a small batch of offline data, DeePO achieves guaranteed global convergence with a fraction of the data required by traditional RL methods.

Methodology

DeePO leverages a "data-driven closed-loop parameterization" rather than treating the system as a black box. The approach is built on several key pillars:

  1. The Fundamental Lemma: Using behavioral systems theory, the authors demonstrate that a batch of "persistently exciting" (PE) raw data contains all necessary information to describe a linear system's behavior without explicit identification of system matrices.
  2. Direct Gradient Computation: System behavior is captured in data matrices, allowing for the derivation of exact policy gradients directly from offline data.
  3. Projected Gradient Descent: Because the policy must satisfy linear constraints to remain feasible within the data-defined subspace, the algorithm employs a projected gradient update.
Geometric interpretation of the projected gradient descent in DeePO, showing the relationship between the initial policy $G^0$, the optimal policy $G^*$, and the projection onto the null space $\mathcal{N}(D_-)$.

To ensure global convergence, the authors relate the non-convex PO problem to an equivalent convex program. This establishes a projected gradient dominance property, guaranteeing that the algorithm reaches the global optimum despite the constraints imposed by the data.

Key Insights

  • Bridging the Gap: DeePO proves that direct RL-style methods can achieve the efficiency of system identification by correctly structuring the problem using behavioral data.
  • Implicit Regularization: The algorithm can exhibit regularized behavior (improving robustness or handling noise) depending on initialization, even without explicit penalty terms. This helps explain the strong practical performance of simple gradient methods.

Experimental Results

The framework was validated using synthetic datasets from simulated Linear Time-Invariant (LTI) systems. In a numerical example with 4 states and 2 inputs, DeePO achieved a 0.01 relative error using a batch of only $T=10$ state-input pairs—the theoretical minimum for persistent excitation. In contrast, state-of-the-art zeroth-order PO methods often require upwards of $10^5$ trajectories for similar accuracy.

Convergence performance of DeePO compared to certainty-equivalence (CE) and robust regularization methods, illustrating linear convergence on a log-scale.

While theoretical proofs currently guarantee sublinear convergence, simulations consistently demonstrate linear convergence (as seen in the straight-line trajectories on the log-scale plot above), suggesting the method may possess even stronger properties than currently proven.

Conclusion and Future Work

DeePO successfully bridges the gap between data-efficient control and simple policy optimization for LQR systems. Future research will focus on extending these results to non-linear systems and developing a "strongly convex" reparameterization to theoretically confirm the linear convergence observed in practice. Additionally, the authors aim to adapt DeePO for real-time online control applications.

Summary updated

BibTeX
@inproceedings{zhao2023data,
  author = {Feiran Zhao and Florian Dörfler and Keyou You},
  title = {{Data-Enabled Policy Optimization for the Linear Quadratic Regulator}},
  booktitle = {CDC},
  year = {2023},
  doi = {10.1109/cdc49753.2023.10383470},
}
Figure