Optimistic Distributionally Robust Policy Optimization

Jun Song, Chaoyue Zhao

Abstract

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), as the widely employed policy based reinforcement learning (RL) methods, are prone to converge to a sub-optimal solution as they limit the policy representation to a particular parametric distribution class. To address this issue, we develop an innovative Optimistic Distributionally Robust Policy Optimization (ODRPO) algorithm, which effectively utilizes Optimistic Distributionally Robust Optimization (DRO) approach to solve the trust region constrained optimization problem without parameterizing the policies. Our algorithm improves TRPO and PPO with a higher sample efficiency and a better performance of the final policy while attaining the learning stability. Moreover, it achieves a globally optimal policy update that is not promised in the prevailing policy based RL algorithms. Experiments across tabular domains and robotic locomotion tasks demonstrate the effectiveness of our approach.

Introduction

Most modern policy-based reinforcement learning (RL) methods, such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), rely on a critical assumption: the policy must belong to a specific parametric distribution class, such as a Gaussian or Beta distribution. While this simplifies the optimization landscape, it creates a "parametric trap" or a "glass ceiling." If the optimal behavior for an agent cannot be accurately described by the chosen distribution shape, these algorithms will inevitably converge to a sub-optimal solution. Furthermore, TRPO and PPO often rely on linear or quadratic approximations to solve their optimization problems, which can lead to unstable updates or inefficient use of samples.

To overcome these limitations, we introduce Optimistic Distributionally Robust Policy Optimization (ODRPO). This framework shifts the focus from optimizing parameters ($\theta$) to optimizing the probability distribution ($\pi$) itself, allowing the policy to take any shape necessary to solve the task.

Methodology

ODRPO leverages the principles of Distributionally Robust Optimization (DRO) but with an "optimistic" twist. While traditional DRO is often pessimistic—seeking decisions that perform best under the worst-case scenario—ODRPO looks for the best (most optimistic) distribution within an "ambiguity set" (the trust region) to maximize expected rewards.

Non-Parametric Updates and Dual Formulation

By considering all admissible distributions rather than restricting the policy to a pre-defined shape, ODRPO can find the exact globally optimal policy update within the allowed trust region. The authors derive the mathematical dual of this optimization problem, providing closed-form solutions for the optimal update. This avoids the need for the approximations (like Conjugate Gradient) used in TRPO that may fail to find the true best step.

Trust Region Metrics

The paper explores two distinct ways to define the "ambiguity set" or trust region:

  1. KL Divergence: This leads to an exponential-style update and generally exhibits faster convergence because the uncertainty set shrinks exponentially with more samples.
  2. Wasserstein Distance: Based on the theory of optimal transport, this metric is particularly effective for discrete action spaces, though the uncertainty set shrinks more slowly (polynomially) compared to KL divergence.

Experimental Results

Robotic Locomotion and Continuous Control

The effectiveness of ODRPO was evaluated against TRPO, PPO, and A2C across high-dimensional robotic locomotion tasks in the MuJoCo and OpenAI Gym environments. ODRPO consistently achieved superior performance, often outperforming the best baseline results by an average of 75%.

Average return curves for various continuous control tasks, comparing ODRPO variants against standard baselines.

As seen in the performance curves, ODRPO-KL (blue) frequently reaches higher asymptotic performance and demonstrates more stable learning trajectories than PPO or TRPO. In environments like Walker2d and Hopper, the gap in final average return is particularly pronounced.

Sample Efficiency

One of the most significant advantages of ODRPO is its sample efficiency. Because it finds the globally optimal update for each step, it requires significantly fewer environment interactions to reach performance milestones.

Table showing the number of timesteps (scaled) required to reach specific performance thresholds across different environments.

For example, in the Walker2d task, ODRPO reached the target threshold in 45 units of time, whereas TRPO required 112 units—a more than 2x improvement in efficiency. Similar trends are observed in Hopper and Reacher, where ODRPO reaches thresholds in a fraction of the time required by PPO or TRPO.

Tabular Domains and the Parametric Trap

In the "Taxi" tabular environment, the limitations of parametric models were starkly revealed. TRPO failed to reach the goal entirely (0% success rate) because its parametric constraints prevented it from representing the necessary policy. In contrast, the non-parametric ODRPO-KL achieved a 99.5% success rate, proving that distributional flexibility is a critical advantage in complex state spaces.

Conclusion

ODRPO represents a significant step forward in policy-based RL by breaking free from parametric constraints. By providing a globally optimal update within a distributionally robust framework, it achieves higher sample efficiency and better final performance while maintaining the stability that makes trust-region methods desirable. Future research will focus on extending this non-parametric approach to off-policy frameworks and exploring its scalability in extremely high-dimensional action spaces.

Summary updated

Abstract

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), as the widely employed policy based reinforcement learning (RL) methods, are prone to converge to a sub-optimal solution as they limit the policy representation to a particular parametric distribution class. To address this issue, we develop an innovative Optimistic Distributionally Robust Policy Optimization (ODRPO) algorithm, which effectively utilizes Optimistic Distributionally Robust Optimization (DRO) approach to solve the trust region constrained optimization problem without parameterizing the policies. Our algorithm improves TRPO and PPO with a higher sample efficiency and a better performance of the final policy while attaining the learning stability. Moreover, it achieves a globally optimal policy update that is not promised in the prevailing policy based RL algorithms. Experiments across tabular domains and robotic locomotion tasks demonstrate the effectiveness of our approach.
BibTeX
@misc{song2020optimistic,
  author = {Jun Song and Chaoyue Zhao},
  title = {{Optimistic Distributionally Robust Policy Optimization}},
  year = {2020},
  eprint = {2006.07815},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.07815},
}

Loading related content...

Figure