On Proximal Policy Optimization's Heavy-Tailed Gradients

Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, J Zico Kolter, Zachary Chase Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, Pradeep Kumar Ravikumar

TL;DR

This paper investigates the "heavy-tailed" nature of gradients in Proximal Policy Optimization (PPO), a popular reinforcement learning algorithm. They discover that these heavy tails, particularly in the actor network's gradients, are a significant source of instability and that PPO's clipping heuristics are primarily designed to mitigate this issue. By replacing these heuristics with a robust estimation method (GMOM), they achieve comparable performance to standard PPO.

What Problem Does This Paper Address?

Reinforcement Learning (RL) algorithms, especially those using policy gradients like PPO, often employ various "heuristics" such as gradient clipping and loss clipping. These techniques are crucial for stabilizing training and achieving good performance. However, the exact reasons behind their effectiveness, and the underlying phenomena they address, haven't been fully understood. This paper posits that these heuristics are reminiscent of methods used in robust statistics to handle "heavy-tailed" data distributions, where extreme values (outliers) are more common than in a standard Gaussian distribution. The core problem this paper addresses is to empirically characterize the heavy-tailed nature of PPO's gradients, understand its sources, and demonstrate how the existing heuristics combat it, proposing a more principled alternative.

Methods

The authors conduct a detailed empirical study to analyze the gradients of PPO. They employ several statistical measures to quantify heavy-tailedness, including:

  • Kurtosis: A measure of the "tailedness" of a distribution, where higher kurtosis indicates heavier tails.
  • Alpha-index estimator: Derived from a-stable distributions, a lower alpha-index signifies heavier tails.
  • Anderson-Darling test: Used to assess Gaussianity, with a lower acceptance rate indicating non-Gaussian (potentially heavy-tailed) behavior.

They analyze gradients at different stages of training and under varying degrees of "off-policyness" (when the agent learns from data generated by an older policy). A key methodological contribution is the proposal of Geometric Median-of-Means (GMOM), a robust aggregation method from statistics, as a replacement for PPO's clipping heuristics. They integrate GMOM into a simplified PPO variant (PPO-NOCLIP) to create ROBUST-PPO-NOCLIP.

Datasets

The experiments and evaluations were conducted on a battery of MuJoCo continuous control tasks. The paper does not introduce new benchmarks.

Key Insights and Ideas

  • PPO's heuristics are for heavy tails: The paper strongly suggests that PPO's clipping mechanisms are not just arbitrary tricks but are fundamentally designed to counteract the instability caused by heavy-tailed gradients.
  • Off-policyness exacerbates heavy tails: As the agent deviates further from its past behavior (i.e., becomes more "off-policy"), the gradients become significantly more heavy-tailed, especially in the actor network.
  • Likelihood ratios are a major culprit: During off-policy learning, the "likelihood ratios" (which measure how much the current policy differs from the behavior policy) are identified as a primary source of heavy-tailedness.
  • Advantages are key on-policy: During on-policy steps, the "advantage estimates" (which represent how much better an action was than the average) are the main contributors to heavy-tailed gradients.
  • Negative advantages are particularly problematic: Histograms reveal that negative advantages are disproportionately heavy-tailed, contributing significantly to overall gradient instability.
  • GMOM offers a robust alternative: Replacing PPO's clipping heuristics with the GMOM robust estimator allows for comparable performance, supporting the hypothesis that addressing heavy-tailedness directly is effective.

Key Results

  • Increased heavy-tailedness with training: Kurtosis analysis shows a consistent increase in heavy-tailedness of PPO gradients (both actor and critic) as training progresses.
  • Off-policy impact: Heavy-tailedness in gradients increases significantly with off-policy steps, with actor gradients being more affected.
  • GMOM matches PPO: The proposed ROBUST-PPO-NOCLIP method, using GMOM instead of clipping, achieves performance comparable to standard PPO (with all heuristics enabled) across several MuJoCo tasks.
  • Advantage clipping improves performance: Clipping outlier advantages, particularly negative ones, leads to improved performance on a majority of MuJoCo tasks.
  • Likelihood ratio clipping is crucial: Increasing off-policy epochs in PPO-NOCLIP leads to performance degradation and instability, highlighting the importance of handling heavy-tailed likelihood ratios.

Limitations and Future Work

  • Heuristic nature of estimators: While kurtosis and alpha-index are used as heuristic measures, their limitations in finite sample settings are acknowledged.
  • Tuning GMOM: While GMOM replaces clipping, it still requires hyperparameter tuning (e.g., number of blocks).
  • Future work: The authors suggest extending this analysis to other RL algorithms like deep Q-learning. They also believe their findings on advantage estimates can guide the development of more stable policy gradient algorithms.

Summary updated

Abstract

Modern policy gradient algorithms, notably Proximal Policy Optimization (PPO), rely on an arsenal of heuristics, including loss clipping and gradient clipping, to ensure successful learning. These heuristics are reminiscent of techniques from robust statistics, commonly used for estimation in outlier-rich ("heavy-tailed") regimes. In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function. We demonstrate pronounced heavy-tailedness of the gradients, specifically for the actor network, which increases as the current policy diverges from the behavioral one (i.e., as the agent goes further off policy). Further examination implicates the likelihood ratios and advantages in the surrogate reward as the main sources to the observed heavy-tailedness. Subsequently, we study the effects of the standard PPO clipping heuristics, demonstrating how these tricks primarily serve to offset heavy-tailedness in gradients. Motivated by these connections, we propose incorporating GMOM (a high-dimensional robust estimator) into PPO as a substitute for three clipping tricks, achieving performance close to PPO (with all heuristics enabled) on a battery of MuJoCo continuous control tasks.
BibTeX
@inproceedings{garg2021proximal,
  author = {Saurabh Garg and Joshua Zhanson and Emilio Parisotto and Adarsh Prasad and J Zico Kolter and Zachary Chase Lipton and Sivaraman Balakrishnan and Ruslan Salakhutdinov and Pradeep Kumar Ravikumar},
  title = {{On Proximal Policy Optimization's Heavy-Tailed Gradients}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure