Decaying Clipping Range in Proximal Policy Optimization

Mónika Farsang, Luca Szegletes

TL;DR

This paper proposes a simple but effective modification to Proximal Policy Optimization (PPO) by replacing the constant clipping range with linearly or exponentially decaying values. This approach encourages broader exploration at the start of training and enforces stricter policy updates toward the end, leading to significant performance gains in complex robotic locomotion tasks.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), Policy Gradient methods often suffer from instability: a single large update can move the policy into a region of the parameter space from which it can never recover. Proximal Policy Optimization (PPO) solved this by introducing a "clipping" mechanism. It restricts how much the policy can change in one step by clipping the probability ratio between the new and old policy within a range (typically $[1-\epsilon, 1+\epsilon]$, where $\epsilon$ is usually 0.2).

However, a constant clipping range is a "one-size-fits-all" solution. Early in training, a tight range might stifle the exploration needed to find good strategies. Conversely, late in training, that same range might be too permissive, preventing the agent from fine-tuning its policy into a truly optimal state. The authors seek to address this "insufficient exploration" and "weak restriction" trade-off without adding heavy computational overhead or complex new hyperparameters.

Methods

The authors introduce two strategies to vary the clipping parameter $\epsilon$ over the course of training:

  1. Linear Decay: The clipping range starts at an initial value $\epsilon_0$ and decreases linearly to zero as training progresses toward the maximum timestep $T$.
  2. Exponential Decay: The range decays more slowly using an exponential function ($\epsilon_t^{exp} = \alpha^{100 \frac{t}{T}} \epsilon_0$), where $\alpha$ is a decay constant (set to 0.99 in their tests).

The core PPO algorithm remains unchanged otherwise, maintaining its simplicity and scalability. The researchers implemented these changes using the Stable Baselines3 (SB3) framework, ensuring the results are comparable to standard, high-quality implementations.

Datasets

The researchers evaluated their approach across six environments of increasing complexity:

  • OpenAI Gym (Classical Control): Cart-Pole, Pendulum, and Acrobot. These serve as benchmarks for basic stability and convergence.
  • PyBullet (Robotic Locomotion): Hopper, Walker2D, and Half-Cheetah. These are high-dimensional tasks where the agent must learn complex gait patterns to move forward.

Key Insights and Ideas

  • Training Wheels for Agents: High clipping values at the start act like "training wheels," allowing the agent to make larger leaps in logic to explore the environment. As the agent gains "experience," the decay "tightens" the constraints, forcing the agent to settle into a stable, high-performing policy.
  • The Clipping "U-Curve": The authors analyzed how often clipping actually occurs (Fig 2). They found that clipping is frequent at the start (due to poor initial performance) and rises again at the end (because the decay makes the "allowable" update window much smaller). This confirms that the decaying range is actively shaping the learning process in the final stages.
  • Complexity Sensitivity: The benefits of decaying clipping are most pronounced in complex environments. In simple tasks like Cart-Pole, all methods eventually hit the maximum score, but in high-dimensional tasks like Half-Cheetah, the strategy used makes a massive difference in the final reward.

Key Results

The results demonstrate that decaying clipping ranges are a "considerable alternative" to the constant method:

  • Complex Tasks: In the Walker2D environment, the exponential decay strategy achieved a final score of 1356.59, vastly outperforming the constant clipping score of 984.99.
  • Half-Cheetah: Exponential decay again led the pack with a score of 1982.20 compared to the constant version's 1891.02.
  • Simple Tasks: For Cart-Pole, all methods reached the perfect score of 500.00. For the Pendulum, the linear decay showed the best "all training" average, but exponential decay achieved the best final performance in the last 100 episodes.
  • Efficiency: These improvements were achieved without adding extra computational load, making it a "free" upgrade for PPO users.

Limitations and Future Work

While the results are promising, the paper has a few areas for further exploration:

  • Hyperparameter Sensitivity: The study primarily uses a fixed $\alpha$ for exponential decay and a fixed $\epsilon_0$. It is unclear how sensitive the results are to these specific choices across different types of environments.
  • Adaptive Decay: The decay is currently a function of time (timesteps). A more advanced version might adapt the clipping range based on the agent's actual performance or the variance of the advantage estimates.
  • Scope: The study focuses on continuous control and classical tasks; testing on discrete action spaces or pixel-based inputs (like Atari) would further validate the robustness of the approach.

Reference: Farsang, M., & Szegletes, L. (2021). Decaying Clipping Range in Proximal Policy Optimization. arXiv:2102.10456v3.

Summary updated

Abstract

Proximal Policy Optimization (PPO) is among the most widely used algorithms in reinforcement learning, which achieves state-of-the-art performance in many challenging problems. The keys to its success are the reliable policy updates through the clipping mechanism and the multiple epochs of minibatch updates. The aim of this research is to give new simple but effective alternatives to the former. For this, we propose linearly and exponentially decaying clipping range approaches throughout the training. With these, we would like to provide higher exploration at the beginning and stronger restrictions at the end of the learning phase. We investigate their performance in several classical control and locomotive robotic environments. During the analysis, we found that they influence the achieved rewards and are effective alternatives to the constant clipping method in many reinforcement learning tasks.
BibTeX
@inproceedings{farsang2021decaying,
  author = {Mónika Farsang and Luca Szegletes},
  title = {{Decaying Clipping Range in Proximal Policy Optimization}},
  booktitle = {SACI},
  year = {2021},
  url = {https://openreview.net/forum?id=33LBSqnony},
}

Loading related content...

Figure