Introduction
Direct policy search is the engine behind modern reinforcement learning (RL) and continuous control. However, its theoretical properties in nonsmooth robust control synthesis have remained largely elusive. The optimal $\mathcal{H}_\infty$ control framework, which minimizes the closed-loop $\mathcal{H}_\infty$ norm, is the fundamental paradigm for robust control—guaranteeing performance even against adversarial disturbances. This paper bridges the gap between nonsmooth optimization theory and robust control, proving that direct policy search can globally solve the classic $\mathcal{H}_\infty$ state-feedback robust control problem.
The Challenge of $\mathcal{H}_\infty$ Control
In control theory, the Linear Quadratic Regulator (LQR) handles stochastic noise (the "average" case), while $\mathcal{H}_\infty$ control addresses worst-case adversarial disturbances. While direct policy search has been successfully applied to LQR, applying it to $\mathcal{H}_\infty$ control presents two major hurdles:
- Nonconvexity: The feasible set of stabilizing controllers forms a complex, nonconvex shape.
- Nonsmoothness: Unlike LQR, the $\mathcal{H}_\infty$ objective function is not differentiable everywhere. It contains "kinks" where standard gradients do not exist, causing traditional gradient descent methods to fail or get trapped.
Until this work, there was no theoretical guarantee that an RL-style algorithm wouldn't get stuck in a local trap or fail entirely at a nonsmooth point when optimizing for worst-case robustness.
Methodological Breakthroughs
To overcome these challenges, the authors leverage advanced nonsmooth optimization techniques and classic control theory:
- Goldstein Subdifferential: Because the objective function is nonsmooth, a single gradient is insufficient. The authors employ the Goldstein subdifferential, which evaluates the "average" gradient within a small neighborhood of the current policy. This yields a reliable descent direction even at non-differentiable points.
- Coerciveness Analysis: The authors prove that the $\mathcal{H}_\infty$ cost function is "coercive." This means the cost approaches infinity as the controller nears the boundary of instability. Consequently, any descent algorithm initialized with a stabilizing policy will naturally remain within the "safe" set of stabilizing controllers, ensuring all sublevel sets are compact.
- Bounded Real Lemma and LMIs: By utilizing classic control tools like Linear Matrix Inequalities (LMIs) and the Bounded Real Lemma, the authors re-parameterize the problem to reveal that the underlying geometry of the $\mathcal{H}_\infty$ cost function is highly structured and better behaved than its nonconvexity suggests.
Key Findings and Theoretical Guarantees
- Stationary Points are Global Minima: The most significant theoretical insight is that for the $\mathcal{H}_\infty$ state-feedback problem, every Clarke stationary point (a generalized zero-gradient point for nonsmooth functions) is a global minimum. This effectively eliminates the risk of algorithms getting trapped in local optima.
- Global Convergence Guarantees: The paper provides the first formal proof that Goldstein’s subgradient method and its implementable variants (such as Gradient Sampling) are guaranteed to stay within the nonconvex feasible set and converge to the global optimal $\mathcal{H}_\infty$ solution.
- Complexity Bounds: The authors establish finite-time complexity bounds, showing that finding a $(\delta, \epsilon)$-stationary point requires roughly $\mathcal{O}(1/\epsilon^3)$ or $\mathcal{O}(1/\epsilon^4)$ iterations, depending on the specific algorithm variant.
- Empirical Validation: Using synthetic Linear Time-Invariant (LTI) systems, simulations demonstrate that even "model-free" versions of these algorithms—which rely only on observed costs rather than known system equations—successfully track the performance of model-based methods and converge to the global optimum.
Limitations and Future Directions
While this work provides a rigorous theoretical foundation for using RL in safety-critical applications, several open challenges remain:
- State-Feedback Constraint: The current framework assumes "state-feedback," where the controller has access to the exact system state. Extending this to noisy "output-feedback" (sensor-based) control is a significantly harder, yet vital, next step.
- Linear Systems: The theoretical guarantees are currently limited to linear systems. Generalizing these global convergence results to nonlinear robust control remains a major frontier.
- Optimality Gap Complexity: While the paper bounds the rate at which the algorithm finds a stationary point, the rate at which the actual optimality gap ($J(K) - J^*$) closes requires further investigation.
Summary updated