Abstract
Proximal Policy Optimization (PPO) is among the most widely used algorithms in reinforcement learning, which achieves state-of-the-art performance in many challenging problems. The keys to its success are the reliable policy updates through the clipping mechanism and the multiple epochs of minibatch updates. The aim of this research is to give new simple but effective alternatives to the former. For this, we propose linearly and exponentially decaying clipping range approaches throughout the training. With these, we would like to provide higher exploration at the beginning and stronger restrictions at the end of the learning phase. We investigate their performance in several classical control and locomotive robotic environments. During the analysis, we found that they influence the achieved rewards and are effective alternatives to the constant clipping method in many reinforcement learning tasks.
Introduction
In Reinforcement Learning (RL), Policy Gradient methods often suffer from instability: a single large update can move the policy into a region of the parameter space from which it can never recover. Proximal Policy Optimization (PPO) addressed this by introducing a "clipping" mechanism. It restricts how much the policy can change in one step by clipping the probability ratio between the new and old policy within a range (typically $[1-\epsilon, 1+\epsilon]$, where $\epsilon$ is usually 0.2).
However, a constant clipping range is a "one-size-fits-all" solution. Early in training, a tight range might stifle the exploration needed to find good strategies. Conversely, late in training, that same range might be too permissive, preventing the agent from fine-tuning its policy into a truly optimal state. This research addresses this "insufficient exploration" and "weak restriction" trade-off by introducing a decaying clipping range.
Methods
The proposed modification replaces the constant clipping parameter $\epsilon$ with values that decay over the course of training. Two strategies are introduced:
- Linear Decay: The clipping range starts at an initial value $\epsilon_0$ and decreases linearly to zero as training progresses toward the maximum timestep $T$.
- Exponential Decay: The range decays more slowly using an exponential function ($\epsilon_t^{exp} = \alpha^{100 \frac{t}{T}} \epsilon_0$), where $\alpha$ is a decay constant (set to 0.99 in tests).
The core PPO algorithm remains unchanged otherwise, maintaining its simplicity and scalability. The researchers implemented these changes using the Stable Baselines3 (SB3) framework, ensuring the results are comparable to standard, high-quality implementations.
Experimental Setup
The approach was evaluated across six environments of increasing complexity, ranging from classical control to high-dimensional robotic locomotion.
- OpenAI Gym (Classical Control): Cart-Pole, Pendulum, and Acrobot. These serve as benchmarks for basic stability and convergence.
- PyBullet (Robotic Locomotion): Hopper, Walker2D, and Half-Cheetah. These are high-dimensional tasks where the agent must learn complex gait patterns to move forward.
Key Insights
- Training Wheels for Agents: High clipping values at the start act like "training wheels," allowing the agent to make larger leaps in logic to explore the environment. As the agent gains experience, the decay tightens the constraints, forcing the agent to settle into a stable, high-performing policy.
- The Clipping "U-Curve": Analysis of clipping frequency showed that clipping is frequent at the start (due to poor initial performance) and rises again at the end (because the decay makes the allowable update window much smaller). This confirms that the decaying range actively shapes the learning process in the final stages.
- Complexity Sensitivity: The benefits of decaying clipping are most pronounced in complex environments. In simple tasks like Cart-Pole, all methods eventually hit the maximum score, but in high-dimensional tasks like Half-Cheetah, the strategy used makes a massive difference in the final reward.
Results
The results demonstrate that decaying clipping ranges are a considerable alternative to the constant method:
- Complex Tasks: In the Walker2D environment, the exponential decay strategy achieved a final score of 1356.59, vastly outperforming the constant clipping score of 984.99.
- Half-Cheetah: Exponential decay again led the pack with a score of 1982.20 compared to the constant version's 1891.02.
- Simple Tasks: For Cart-Pole, all methods reached the perfect score of 500.00. For the Pendulum, the linear decay showed the best "all training" average, but exponential decay achieved the best final performance in the last 100 episodes.
- Efficiency: These improvements were achieved without adding extra computational load, making it a "free" upgrade for PPO users.
Limitations and Future Work
While the results are promising, several areas warrant further exploration. The study primarily uses fixed parameters for decay ($\alpha$ and $\epsilon_0$); sensitivity analysis across more environments would be beneficial. Future work could explore Adaptive Decay, where the clipping range adapts based on the agent's actual performance or the variance of advantage estimates, rather than just being a function of time. Additionally, expanding the scope to discrete action spaces or pixel-based inputs (like Atari) would further validate the robustness of the approach.
Reference: Farsang, M., & Szegletes, L. (2021). Decaying Clipping Range in Proximal Policy Optimization. arXiv:2102.10456v3.
Summary updated