Beyond the Boundaries of Proximal Policy Optimization

Charlie B. Tan, Edan Toledo, Benjamin Ellis, Jakob Nicolaus Foerster, Ferenc HuszárSubmission to ICLR 2025 · Rejected

TL;DR

Proximal Policy Optimization (PPO) implicitly performs gradient ascent in an "outer loop" with a fixed learning rate of 1. By reframing PPO to decouple update estimation from update application, Outer-PPO allows researchers to tune outer learning rates, apply Nesterov momentum, or warm-start inner optimization. Across continuous control (Brax) and discrete environments (Jumanji), simply scaling the outer step size yields a 5–10% performance boost over an aggressively tuned baseline without adding computational overhead.

What Problem Does This Paper Address?

Standard PPO alternates between two phases: gathering trajectories with a behavior policy, and optimizing a surrogate objective over multiple minibatches within a "trust region" defined by clipping threshold $\epsilon$. Once the inner optimization finds candidate parameters $\theta^*_k$, PPO simply sets the next behavior policy to $\theta_{k+1} = \theta^*_k$.

This couples two separate concepts:

  1. Direction estimation: Finding where to improve the policy using surrogate objectives.
  2. Step execution: Deciding how far or fast to actually move in parameter space.

This setup makes three implicit design assumptions that are rarely questioned:

  1. A unity step size ($\sigma = 1$) on the parameter update is always optimal.
  2. Each outer iteration should be independent, ignoring trajectory history or momentum.
  3. The inner optimization loop must always initialize strictly at the current behavior parameters $\theta_k$.

Methods

The authors reframe PPO as an inner-outer optimization process. The inner loop solves the surrogate objective to produce an outer gradient: $$g_k^O = \text{PPOIteration}(\theta_k) - \theta_k$$ Standard PPO is simply outer gradient ascent with learning rate $\sigma = 1$: $\theta_{k+1} = \theta_k + 1 \cdot g_k^O$.

Under the Outer-PPO framework, arbitrary optimizers can act on $g_k^O$:

  • Outer Learning Rates (Outer-LR): Sets $\theta_{k+1} = \theta_k + \sigma g_k^O$. When $\sigma < 1$, it acts as conservative interpolation. When $\sigma > 1$, it takes larger steps in the direction estimated by a well-conditioned, clipped surrogate without dangerously widening the clipping parameter $\epsilon$.
  • Outer-Nesterov PPO: Tracks momentum $m_k = \mu m_{k-1} + g_k^O$ across outer iterations and applies a Nesterov update step before the next rollout. This smooths out high-variance policy gradient updates across batches.
  • Biased Initialization: Applies momentum before data collection and surrogate optimization, initializing the inner loop at $\theta_k + \alpha m_{k-1}$ (reminiscent of conjugate gradient initialization in Hessian-free optimization).

Datasets

The authors evaluated on 14 tasks across three diverse reinforcement learning benchmark suites implemented in JAX:

  • Brax (6 continuous control tasks): ant, halfcheetah, hopper, humanoid, humanoidstandup, walker2d.
  • Jumanji (4 combinatorial / discrete control tasks): game_2048, maze, rubiks_cube, snake.
  • MinAtar (4 arcade-style discrete tasks): asterix, breakout, freeway, space_invaders.

No new environments were introduced; instead, the authors open-sourced extensive hyperparameter sweep databases (tuning 11 base parameters across 2,400 agent runs per task) to ensure strong, fair baselines.

Key Insights and Ideas

  • Trust region clipping and step length are fundamentally different: In standard PPO, taking larger policy updates requires increasing $\epsilon$, which risks moving into unrepresentative policy regions where importance weights become unreliable. Outer-LR decouples this: a moderate $\epsilon$ reliably finds the direction, while $\sigma > 1$ steps aggressively along it.
  • The unity learning rate is often sub-optimal: Across Brax and Jumanji, optimal outer learning rates were frequently significantly larger than 1 (reaching up to $\sigma = 2.3$ on snake and $\sigma = 2.1$ on humanoidstandup).
  • Noise smoothing via momentum helps: Outer Nesterov momentum provided consistent gains across environments, acting as a temporal filter against single-batch rollout noise.
  • Negligible compute overhead: Outer-PPO only adds element-wise vector scaling and additions once per outer iteration. Wall-clock runtimes are virtually identical to vanilla PPO.

Key Results

Evaluation was conducted using 64 independent seeds per task over 10 million environment transitions:

  • Outer-LR: Achieved a 5–10% aggregate improvement over the tuned PPO baseline across Brax and Jumanji on interquartile mean (IQM), mean, and median returns. Probability of improvement over baseline was $>0.6$ on Brax and $>0.7$ on Jumanji.
  • Outer-Nesterov: Also produced statistically significant gains across all summary metrics on Jumanji and Brax (except Brax median).
  • Biased Initialization: Yielded moderate, statistically significant gains on Jumanji, but was inconsistent on Brax due to high variance.
  • MinAtar Plateau: None of the methods outperformed the baseline on MinAtar, largely because the heavily tuned PPO baseline already operated near the empirical ceiling (normalized returns $>0.9$).

Limitations and Future Work

  • Fixed Budget Horizon: All experiments were capped at $10^7$ environment steps; asymptotic performance in large-scale, data-rich regimes remains unmeasured.
  • Hyperparameter Co-Optimization: Most Outer-PPO sweeps froze the base PPO hyperparameters discovered in earlier trials rather than performing an exhaustive joint co-optimization across all 12–13 dimensions.
  • Interaction with Inner Learning Rate Decay: All experiments employed linear learning rate annealing on the inner Adam optimizer, which forces outer gradients toward zero over time; the dynamics of this interaction were not isolated.
  • Future Directions: Applying adaptive optimizers (e.g., outer Adam, RMSprop) or integrating Outer-PPO with KL-penalized or discovered policy optimization (DPO) objectives.

Summary updated

Abstract

Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumaji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumaji, given the same hyperparameter tuning budget.
BibTeX
@misc{tan2024beyond,
  author = {Charlie B. Tan and Edan Toledo and Benjamin Ellis and Jakob Nicolaus Foerster and Ferenc Huszár},
  title = {{Beyond the Boundaries of Proximal Policy Optimization}},
  year = {2024},
  eprint = {2411.00666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00666},
}
Figure