Global Convergence of Direct Policy Search for State-Feedback $\mathcal{H}_\infty$ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential

Xingang Guo, Bin Hu

Introduction

Direct policy search is the engine behind modern reinforcement learning (RL) and continuous control. However, its theoretical properties in nonsmooth robust control synthesis have remained largely elusive. The optimal $\mathcal{H}_\infty$ control framework, which minimizes the closed-loop $\mathcal{H}_\infty$ norm, is the fundamental paradigm for robust control—guaranteeing performance even against adversarial disturbances. This paper bridges the gap between nonsmooth optimization theory and robust control, proving that direct policy search can globally solve the classic $\mathcal{H}_\infty$ state-feedback robust control problem.

The Challenge of $\mathcal{H}_\infty$ Control

In control theory, the Linear Quadratic Regulator (LQR) handles stochastic noise (the "average" case), while $\mathcal{H}_\infty$ control addresses worst-case adversarial disturbances. While direct policy search has been successfully applied to LQR, applying it to $\mathcal{H}_\infty$ control presents two major hurdles:

  1. Nonconvexity: The feasible set of stabilizing controllers forms a complex, nonconvex shape.
  2. Nonsmoothness: Unlike LQR, the $\mathcal{H}_\infty$ objective function is not differentiable everywhere. It contains "kinks" where standard gradients do not exist, causing traditional gradient descent methods to fail or get trapped.

Until this work, there was no theoretical guarantee that an RL-style algorithm wouldn't get stuck in a local trap or fail entirely at a nonsmooth point when optimizing for worst-case robustness.

Methodological Breakthroughs

To overcome these challenges, the authors leverage advanced nonsmooth optimization techniques and classic control theory:

  • Goldstein Subdifferential: Because the objective function is nonsmooth, a single gradient is insufficient. The authors employ the Goldstein subdifferential, which evaluates the "average" gradient within a small neighborhood of the current policy. This yields a reliable descent direction even at non-differentiable points.
  • Coerciveness Analysis: The authors prove that the $\mathcal{H}_\infty$ cost function is "coercive." This means the cost approaches infinity as the controller nears the boundary of instability. Consequently, any descent algorithm initialized with a stabilizing policy will naturally remain within the "safe" set of stabilizing controllers, ensuring all sublevel sets are compact.
  • Bounded Real Lemma and LMIs: By utilizing classic control tools like Linear Matrix Inequalities (LMIs) and the Bounded Real Lemma, the authors re-parameterize the problem to reveal that the underlying geometry of the $\mathcal{H}_\infty$ cost function is highly structured and better behaved than its nonconvexity suggests.

Key Findings and Theoretical Guarantees

  • Stationary Points are Global Minima: The most significant theoretical insight is that for the $\mathcal{H}_\infty$ state-feedback problem, every Clarke stationary point (a generalized zero-gradient point for nonsmooth functions) is a global minimum. This effectively eliminates the risk of algorithms getting trapped in local optima.
  • Global Convergence Guarantees: The paper provides the first formal proof that Goldstein’s subgradient method and its implementable variants (such as Gradient Sampling) are guaranteed to stay within the nonconvex feasible set and converge to the global optimal $\mathcal{H}_\infty$ solution.
  • Complexity Bounds: The authors establish finite-time complexity bounds, showing that finding a $(\delta, \epsilon)$-stationary point requires roughly $\mathcal{O}(1/\epsilon^3)$ or $\mathcal{O}(1/\epsilon^4)$ iterations, depending on the specific algorithm variant.
  • Empirical Validation: Using synthetic Linear Time-Invariant (LTI) systems, simulations demonstrate that even "model-free" versions of these algorithms—which rely only on observed costs rather than known system equations—successfully track the performance of model-based methods and converge to the global optimum.

Limitations and Future Directions

While this work provides a rigorous theoretical foundation for using RL in safety-critical applications, several open challenges remain:

  • State-Feedback Constraint: The current framework assumes "state-feedback," where the controller has access to the exact system state. Extending this to noisy "output-feedback" (sensor-based) control is a significantly harder, yet vital, next step.
  • Linear Systems: The theoretical guarantees are currently limited to linear systems. Generalizing these global convergence results to nonlinear robust control remains a major frontier.
  • Optimality Gap Complexity: While the paper bounds the rate at which the algorithm finds a stationary point, the rate at which the actual optimality gap ($J(K) - J^*$) closes requires further investigation.

Summary updated

Abstract

Direct policy search has been widely applied in modern reinforcement learning and continuous control. However, the theoretical properties of direct policy search on nonsmooth robust control synthesis have not been fully understood. The optimal $\mathcal{H}_\infty$ control framework aims at designing a policy to minimize the closed-loop $\mathcal{H}_\infty$ norm, and is arguably the most fundamental robust control paradigm. In this work, we show that direct policy search is guaranteed to find the global solution of the robust $\mathcal{H}_\infty$ state-feedback control design problem. Notice that policy search for optimal $\mathcal{H}_\infty$ control leads to a constrained nonconvex nonsmooth optimization problem, where the nonconvex feasible set consists of all the policies stabilizing the closed-loop dynamics. We show that for this nonsmooth optimization problem, all Clarke stationary points are global minimum. Next, we identify the coerciveness of the closed-loop $\mathcal{H}_\infty$ objective function, and prove that all the sublevel sets of the resultant policy search problem are compact. Based on these properties, we show that Goldstein's subgradient method and its implementable variants can be guaranteed to stay in the nonconvex feasible set and eventually find the global optimal solution of the $\mathcal{H}_\infty$ state-feedback synthesis problem. Our work builds a new connection between nonconvex nonsmooth optimization theory and robust control, leading to an interesting global convergence result for direct policy search on optimal $\mathcal{H}_\infty$ synthesis.
BibTeX
@inproceedings{guo2022global,
  author = {Xingang Guo and Bin Hu},
  title = {{Global Convergence of Direct Policy Search for State-Feedback $\textbackslash\{\}mathcal\{H\}\_\textbackslash\{\}infty$ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=NtwEUZE6VcL},
}
Figure