Policy Gradient Method For Robust Reinforcement Learning

Yue Wang, Shaofeng Zou

Abstract

This paper develops the first policy gradient method with global optimality guarantee and complexity analysis for robust reinforcement learning under model mismatch. Robust reinforcement learning is to learn a policy robust to model mismatch between simulator and real environment. We first develop the robust policy (sub-)gradient, which is applicable for any differentiable parametric policy class. We show that the proposed robust policy gradient method converges to the global optimum asymptotically under direct policy parameterization. We further develop a smoothed robust policy gradient method and show that to achieve an $\epsilon$-global optimum, the complexity is $\mathcal{O}(\epsilon^{-3})$. We then extend our methodology to the general model-free setting and design the robust actor-critic method with differentiable parametric policy class and value function. We further characterize its asymptotic convergence and sample complexity under the tabular setting. Finally, we provide simulation results to demonstrate the robustness of our methods.

Introduction: The Sim-to-Real Challenge

Reinforcement Learning (RL) has demonstrated remarkable success in controlled simulations, yet deploying these agents into real-world environments remains a significant hurdle. A policy optimized for a flight simulator may fail catastrophically when faced with real-world wind resistance or mechanical friction—a phenomenon known as model mismatch.

In "Policy Gradient Method For Robust Reinforcement Learning," researchers Yue Wang and Shaofeng Zou present a theoretical breakthrough: the first policy gradient (PG) framework for robust RL that provides both global optimality guarantees and rigorous complexity analysis. This work bridges the gap between the practical utility of PG methods and the theoretical requirements of robust optimization.

Theoretical Foundation: Robust MDPs

To address model mismatch, the authors utilize the framework of Robust Markov Decision Processes (RMDPs). Unlike standard MDPs that assume a single, fixed transition probability, RMDPs consider an "uncertainty set" of possible environments. The goal is to find a policy that maximizes performance in the worst-case scenario within this set.

The paper specifically focuses on the R-contamination uncertainty set. In this model, the environment's transitions follow a nominal distribution most of the time but can be arbitrarily perturbed with a small probability $R$. This provides a mathematically tractable yet powerful way to model unpredictable environmental shifts.

Overcoming Non-Differentiability

A primary challenge in robust RL is that the "worst-case" objective function is often non-differentiable due to the inner minimization (or maximization of loss) over the uncertainty set. Standard gradient descent cannot be applied directly to such "jagged" landscapes. The authors introduce two key innovations to solve this:

  1. Robust Policy Sub-gradient: They derive the Fréchet sub-gradient for the robust value function, allowing policy updates even at points where a traditional gradient does not exist.
  2. Smoothing via LogSumExp (LSE): To establish convergence rates, they develop a smoothed approximation of the robust objective using the LogSumExp operator. This transforms the non-differentiable objective into a smooth function, facilitating efficient optimization.

Global Optimality and Complexity

One of the most significant contributions of this work is proving that robust PG can find the global optimum. The authors demonstrate that under direct policy parameterization, the robust objective satisfies the Polyak-Łojasiewicz (PL) condition. This condition ensures that even if the objective is non-convex, gradient-based methods will not get stuck in sub-optimal local minima.

However, robustness comes with a computational cost. The authors show that the smoothed robust PG method achieves an $\epsilon$-global optimum with a complexity of $\mathcal{O}(\epsilon^{-3})$. While this is slightly slower than the $\mathcal{O}(\epsilon^{-2})$ complexity typical of non-robust PG, it represents a necessary "price of robustness" for guaranteed performance under uncertainty.

Robust Actor-Critic for Model-Free RL

The methodology extends beyond theoretical models to practical, model-free settings. The authors design a Robust Actor-Critic algorithm where:

  • A Robust TD Critic estimates the worst-case value function.
  • An Actor updates the policy parameters based on these robust estimates.

In the tabular setting, they characterize the asymptotic convergence and establish a total sample complexity of $\mathcal{O}(\epsilon^{-7})$ for the model-free approach.

Experimental Results

The authors evaluated their methods on synthetic Garnet problems and the OpenAI Gym Taxi environment. They compared their Robust PG against vanilla PG and Robust Adversarial RL (RARL), a common heuristic that trains an agent against an adversarial neural network.

Performance comparison of robust PG vs. non-robust PG across varying levels of model perturbation R.

As shown in the experimental results, when the perturbation level $R$ is zero (no model mismatch), both methods perform similarly. However, as $R$ increases, the performance of the non-robust PG degrades significantly, while the Robust PG maintains a high and stable reward. Furthermore, the Robust Actor-Critic outperformed RARL on the Taxi task, suggesting that a principled approach to uncertainty sets is more effective than heuristic adversarial training.

Discussion and Future Directions

While this work provides a foundational theory for robust PG, several avenues for future research remain. The current global optimality proofs rely on direct parameterization; extending these guarantees to deep neural network function approximators is a major open challenge. Additionally, while the $R$-contamination model is robust, exploring other uncertainty sets (such as those defined by KL-divergence or Wasserstein distance) could broaden the applicability of these methods. Finally, improving the sample efficiency of the model-free estimators remains a priority for practical deployment in high-dimensional tasks.

Summary updated

Abstract

This paper develops the first policy gradient method with global optimality guarantee and complexity analysis for robust reinforcement learning under model mismatch. Robust reinforcement learning is to learn a policy robust to model mismatch between simulator and real environment. We first develop the robust policy (sub-)gradient, which is applicable for any differentiable parametric policy class. We show that the proposed robust policy gradient method converges to the global optimum asymptotically under direct policy parameterization. We further develop a smoothed robust policy gradient method and show that to achieve an $\epsilon$-global optimum, the complexity is $\mathcal O(\epsilon^{-3})$. We then extend our methodology to the general model-free setting and design the robust actor-critic method with differentiable parametric policy class and value function. We further characterize its asymptotic convergence and sample complexity under the tabular setting. Finally, we provide simulation results to demonstrate the robustness of our methods.
BibTeX
@inproceedings{wang2022policy,
  author = {Yue Wang and Shaofeng Zou},
  title = {{Policy Gradient Method For Robust Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=Xu4cUAdPAe},
}
Figure