Abstract
Direct policy search has been widely applied in modern reinforcement learning and continuous control. However, the theoretical properties of direct policy search on nonsmooth robust control synthesis have not been fully understood. The optimal $H_\infty$ control framework aims at designing a policy to minimize the closed-loop $H_\infty$ norm, and is arguably the most fundamental robust control paradigm. In this work, we show that direct policy search is guaranteed to find the global solution of the robust $H_\infty$ state-feedback control design problem. Notice that policy search for optimal $H_\infty$ control leads to a constrained nonconvex nonsmooth optimization problem, where the nonconvex feasible set consists of all the policies stabilizing the closed-loop dynamics. We show that for this nonsmooth optimization problem, all Clarke stationary points are global minimum. Next, we identify the coerciveness of the closed-loop $H_\infty$ objective function, and prove that all the sublevel sets of the resultant policy search problem are compact. Based on these properties, we show that Goldstein's subgradient method and its implementable variants can be guaranteed to stay in the nonconvex feasible set and eventually find the global optimal solution of the $H_\infty$ state-feedback synthesis problem. Our work builds a new connection between nonconvex nonsmooth optimization theory and robust control, leading to an interesting global convergence result for direct policy search on optimal $H_\infty$ synthesis.
Introduction
In control theory, the $H_\infty$ norm represents the gold standard for robustness, focusing on minimizing the system's response to the "worst-case" disturbance. While the Linear Quadratic Regulator (LQR) framework is well-understood and features a smooth objective landscape, $H_\infty$ synthesis is notoriously difficult for direct optimization. The $H_\infty$ objective function is inherently nonsmooth—possessing "kinks" where derivatives are undefined—and the set of stabilizing policies is nonconvex.
Despite the success of reinforcement learning (RL) in smooth control problems, it has remained an open question whether simple, local search-based methods can find global optima in the $H_\infty$ landscape. This paper provides the first theoretical guarantee that direct policy search can globally solve the state-feedback $H_\infty$ robust control problem, proving that the landscape's lack of smoothness does not preclude global convergence.
Theoretical Landscape Analysis
The authors bridge nonsmooth optimization theory with classical robust control to characterize the optimization landscape of the $H_\infty$ norm. Their analysis yields two critical properties:
1. Stationarity Implies Optimality
By utilizing the "Bounded Real Lemma," the authors reparameterize the $H_\infty$ problem into a convex form using Linear Matrix Inequalities (LMIs). This transformation allows them to prove a landmark result: for the original nonsmooth problem, every Clarke stationary point (a generalized equilibrium point for nonsmooth functions) is a global minimum. In essence, the $H_\infty$ landscape has no spurious local minima; if an algorithm reaches a point where no descent direction exists in the Clarke sense, it has reached the global optimum.
2. Coercivity and Implicit Safety
A major challenge in policy search is ensuring the policy remains within the "stabilizing set." The authors prove that the $H_\infty$ objective function is coercive, meaning the cost approaches infinity as a policy nears the boundary of instability. This property acts as a natural barrier, ensuring that descent-based optimization trajectories are "repelled" from unstable regions and remain within the safe feasible set without requiring complex projection steps.
Algorithmic Framework
With the landscape properties established, the paper analyzes Goldstein’s subgradient method and several practical variants suitable for both model-based and model-free settings:
- Gradient Sampling (GS): This method approximates a descent direction by randomly sampling gradients in a small neighborhood around the current policy.
- Non-derivative Sampling (NS): A "zeroth-order" variant that relies solely on function evaluations. This is particularly relevant for model-free RL, where the underlying system dynamics may be unknown.
- Interpolated Normalized Gradient Descent (INGD): A variant designed with provable finite-time complexity, establishing a rate of $O(1/\delta\epsilon)$ to find $(\delta, \epsilon)$-stationary points.
Numerical Verification
The theoretical findings were validated using Linear Time-Invariant (LTI) systems. The authors tested their algorithms on a benchmark 3-state system and various randomly generated 3x3 and 4x4 systems.
In these simulations, Gradient Sampling, Non-derivative Sampling, and INGD all successfully converged to the same global optimal value (e.g., $J^* \approx 7.3475$ for the primary benchmark). Notably, the model-free version—which uses power iteration to estimate the $H_\infty$ norm from data—closely tracked the performance of model-based methods, demonstrating the viability of RL-based robust control.
Conclusion and Future Directions
This research provides a rigorous foundation for applying direct policy search to robust control. By proving that stationarity implies global optimality and that coercivity ensures stability, the authors demonstrate that $H_\infty$ synthesis is more tractable than previously thought.
However, several challenges remain. The current work is limited to state-feedback control; the output-feedback case, where the full state is not observable, presents a significantly more complex landscape. Additionally, extending these global convergence results to nonlinear systems and further quantifying the convergence rates for the optimality gap ($J(K) - J^*$) are key areas for future investigation.
Summary updated