Introduction
In the field of optimal control, the Linear Quadratic Regulator (LQR) is considered a foundational success story. Despite the non-convexity of its cost function, policy gradient (PG) methods are known to converge globally to the optimal solution. This success is largely attributed to the property of gradient dominance, which ensures that a small gradient implies proximity to the global optimum.
However, the more general Linear Quadratic Gaussian (LQG) problem—which incorporates process noise, measurement noise, and partial observability—has long resisted similar theoretical guarantees. In its classical state-space parameterization, the LQG cost function lacks gradient dominance, and the set of stabilizing controllers can be disjoint, making global optimization via gradient descent notoriously difficult. This paper provides the first theoretical guarantee for solving the LQG problem using policy gradient methods by introducing a novel reparameterization.
The Optimization Landscape of LQG
The primary challenge in applying direct policy search to LQG lies in the parameterization of the controller. Traditional dynamic controllers maintain an internal state estimate (e.g., via a Kalman Filter), leading to an optimization landscape fraught with local optima. Without gradient dominance, gradient-based algorithms can become trapped or fail to converge entirely. This paper addresses this bottleneck by proposing a "landscape engineering" approach: rather than changing the algorithm, the authors change the representation of the controller to reveal a more benign optimization surface.
Reparameterization via History Representation
The core innovation of this work is the use of a History Representation (or "lifting"). Instead of a dynamic state-space model, the control input $u_t$ is defined as a linear function of a finite window of past inputs and outputs:
$$u_t = \sum_{i=1}^p (M_i^{(y)} y_{t-i} + M_i^{(u)} u_{t-i})$$
where $p$ is the history length. This representation "lifts" the problem into a higher-dimensional space. The authors prove that this lifted representation is a bijection to the optimal LQG controller. Crucially, they demonstrate that in this new space, the LQG cost function satisfies both gradient dominance and smoothness properties, enabling global convergence guarantees for policy gradient methods.
Global Convergence and the Role of History Length
The authors provide the first theoretical proof that PG-LQG under history representation converges to the optimal controller at a rate of $O(\log(1/\epsilon))$. A critical parameter in this framework is the history length $p$. While a larger $p$ increases the dimensionality of the optimization problem, it significantly improves the conditioning of the cost function.
As shown in Figure 1, increasing $p$ leads to dramatically faster convergence. For instance, a history length of $p=50$ reaches a low optimality gap orders of magnitude faster than $p=5$. This suggests a fundamental trade-off: longer history lengths facilitate easier optimization but require a longer "warm-up" period in real-time applications to populate the history buffer before the policy can be safely applied.
Model-Free Learning and Robustness
In practical scenarios where the system matrices are unknown, the authors employ a model-free approach using a zeroth-order gradient estimator. This method involves applying small perturbations to the controller parameters and estimating the gradient based on the observed change in cost.
Numerical results on an open-loop unstable system (with 4 states, 2 inputs, and 2 outputs) confirm that the algorithm remains robust. As seen in Figure 3, the algorithm consistently converges toward the near-optimal solution even with measurement noise and gradient estimation errors. While a small residual bias remains due to the nature of zeroth-order estimation in noisy environments, the global trend confirms the effectiveness of the history-based landscape.
Conclusion
By shifting from a dynamic state-space parameterization to a history-based representation, this work transforms the LQG problem into a benign optimization task. This theoretical breakthrough bridges the gap between LQR and LQG in the context of policy gradient methods. Future research will focus on reducing the bias in model-free estimation and extending the framework to multi-task control settings where a single policy must manage multiple related dynamical systems.
Summary updated