TL;DR
Proximal Policy Optimization (PPO) implicitly performs gradient ascent in an "outer loop" with a fixed learning rate of 1. By reframing PPO to decouple update estimation from update application, Outer-PPO allows researchers to tune outer learning rates, apply Nesterov momentum, or warm-start inner optimization. Across continuous control (Brax) and discrete environments (Jumanji), simply scaling the outer step size yields a 5–10% performance boost over an aggressively tuned baseline without adding computational overhead.
What Problem Does This Paper Address?
Standard PPO alternates between two phases: gathering trajectories with a behavior policy, and optimizing a surrogate objective over multiple minibatches within a "trust region" defined by clipping threshold $\epsilon$. Once the inner optimization finds candidate parameters $\theta^*_k$, PPO simply sets the next behavior policy to $\theta_{k+1} = \theta^*_k$.
This couples two separate concepts:
- Direction estimation: Finding where to improve the policy using surrogate objectives.
- Step execution: Deciding how far or fast to actually move in parameter space.
This setup makes three implicit design assumptions that are rarely questioned:
- A unity step size ($\sigma = 1$) on the parameter update is always optimal.
- Each outer iteration should be independent, ignoring trajectory history or momentum.
- The inner optimization loop must always initialize strictly at the current behavior parameters $\theta_k$.
Methods
The authors reframe PPO as an inner-outer optimization process. The inner loop solves the surrogate objective to produce an outer gradient: $$g_k^O = \text{PPOIteration}(\theta_k) - \theta_k$$ Standard PPO is simply outer gradient ascent with learning rate $\sigma = 1$: $\theta_{k+1} = \theta_k + 1 \cdot g_k^O$.
Under the Outer-PPO framework, arbitrary optimizers can act on $g_k^O$:
- Outer Learning Rates (Outer-LR): Sets $\theta_{k+1} = \theta_k + \sigma g_k^O$. When $\sigma < 1$, it acts as conservative interpolation. When $\sigma > 1$, it takes larger steps in the direction estimated by a well-conditioned, clipped surrogate without dangerously widening the clipping parameter $\epsilon$.
- Outer-Nesterov PPO: Tracks momentum $m_k = \mu m_{k-1} + g_k^O$ across outer iterations and applies a Nesterov update step before the next rollout. This smooths out high-variance policy gradient updates across batches.
- Biased Initialization: Applies momentum before data collection and surrogate optimization, initializing the inner loop at $\theta_k + \alpha m_{k-1}$ (reminiscent of conjugate gradient initialization in Hessian-free optimization).
Datasets
The authors evaluated on 14 tasks across three diverse reinforcement learning benchmark suites implemented in JAX:
- Brax (6 continuous control tasks):
ant,halfcheetah,hopper,humanoid,humanoidstandup,walker2d. - Jumanji (4 combinatorial / discrete control tasks):
game_2048,maze,rubiks_cube,snake. - MinAtar (4 arcade-style discrete tasks):
asterix,breakout,freeway,space_invaders.
No new environments were introduced; instead, the authors open-sourced extensive hyperparameter sweep databases (tuning 11 base parameters across 2,400 agent runs per task) to ensure strong, fair baselines.
Key Insights and Ideas
- Trust region clipping and step length are fundamentally different: In standard PPO, taking larger policy updates requires increasing $\epsilon$, which risks moving into unrepresentative policy regions where importance weights become unreliable. Outer-LR decouples this: a moderate $\epsilon$ reliably finds the direction, while $\sigma > 1$ steps aggressively along it.
- The unity learning rate is often sub-optimal: Across Brax and Jumanji, optimal outer learning rates were frequently significantly larger than 1 (reaching up to $\sigma = 2.3$ on
snakeand $\sigma = 2.1$ onhumanoidstandup). - Noise smoothing via momentum helps: Outer Nesterov momentum provided consistent gains across environments, acting as a temporal filter against single-batch rollout noise.
- Negligible compute overhead: Outer-PPO only adds element-wise vector scaling and additions once per outer iteration. Wall-clock runtimes are virtually identical to vanilla PPO.
Key Results
Evaluation was conducted using 64 independent seeds per task over 10 million environment transitions:
- Outer-LR: Achieved a 5–10% aggregate improvement over the tuned PPO baseline across Brax and Jumanji on interquartile mean (IQM), mean, and median returns. Probability of improvement over baseline was $>0.6$ on Brax and $>0.7$ on Jumanji.
- Outer-Nesterov: Also produced statistically significant gains across all summary metrics on Jumanji and Brax (except Brax median).
- Biased Initialization: Yielded moderate, statistically significant gains on Jumanji, but was inconsistent on Brax due to high variance.
- MinAtar Plateau: None of the methods outperformed the baseline on MinAtar, largely because the heavily tuned PPO baseline already operated near the empirical ceiling (normalized returns $>0.9$).
Limitations and Future Work
- Fixed Budget Horizon: All experiments were capped at $10^7$ environment steps; asymptotic performance in large-scale, data-rich regimes remains unmeasured.
- Hyperparameter Co-Optimization: Most Outer-PPO sweeps froze the base PPO hyperparameters discovered in earlier trials rather than performing an exhaustive joint co-optimization across all 12–13 dimensions.
- Interaction with Inner Learning Rate Decay: All experiments employed linear learning rate annealing on the inner Adam optimizer, which forces outer gradients toward zero over time; the dynamics of this interaction were not isolated.
- Future Directions: Applying adaptive optimizers (e.g., outer Adam, RMSprop) or integrating Outer-PPO with KL-penalized or discovered policy optimization (DPO) objectives.
Summary updated