Batch size-invariance for policy optimization

Jacob Hilton, Karl Cobbe, John Schulman

In reinforcement learning, batch size is a critical hyperparameter that is often strictly constrained by hardware limitations, such as GPU memory. Ideally, an algorithm should be "batch size-invariant." This means that if you are forced to halve your batch size, you should be able to formulaically adjust other hyperparameters (like the learning rate) to achieve the exact same learning behavior, simply taking twice as many steps.

While standard Stochastic Gradient Descent (SGD) exhibits this property at small batch sizes, popular policy optimization algorithms like Proximal Policy Optimization (PPO) do not. This paper identifies the architectural bottleneck preventing batch size-invariance in PPO and proposes a simple, effective decoupling mechanism to fix it.

The Root of the Problem: Coupled Policies

PPO relies on an "old" policy to perform two fundamentally different jobs simultaneously:

  1. Importance Sampling: Correcting for off-policy data. This strictly requires the behavior policy—the exact policy that was used to collect the data in the environment.
  2. Update Constraint: Preventing the new policy from changing too rapidly via a KL penalty or clipping. This requires a proximal policy to act as an anchor.

Because standard PPO forces the behavior policy and the proximal policy to be the exact same entity, changing the batch size inadvertently alters the "age" of the proximal policy. This coupling breaks batch size-invariance. Furthermore, it makes PPO notoriously brittle in asynchronous training setups where data might be "stale" (collected several updates ago). Using stale data in standard PPO either introduces bias into the importance sampling or corrupts the trust region penalty.

The Solution: Decoupling via EWMA

The authors propose rewriting the PPO and Phasic Policy Gradient (PPG) objectives to explicitly decouple these two roles. In their modified architecture, $\pi_{behav}$ is used strictly for importance sampling, while a distinct $\pi_{prox}$ is used for clipping and KL penalties.

To implement $\pi_{prox}$ without the massive memory overhead of keeping a buffer of old network states, the authors introduce PPO-EWMA and PPG-EWMA. These variants maintain an Exponentially-Weighted Moving Average (EWMA) of the policy network's weights, updating it after every gradient step. This EWMA network serves as the proximal policy.

To achieve true batch size-invariance when dividing the batch size by a factor of $c$, the authors identified three necessary scaling rules:

  • Optimizer Step Size: Divide the Adam optimizer step size by $\sqrt{c}$ (notably different from standard SGD, which scales linearly).
  • EWMA Decay Rate: Adjust the decay rate so the "age" of the proximal policy remains constant in terms of environment steps.
  • Advantage Normalization: Scale the moving averages used for advantage normalization to keep sample sizes constant.

Key Results

The authors evaluated their methods on the Procgen Benchmark, a suite of 16 procedurally generated RL environments designed to test sample efficiency and generalization.

Staleness Tolerance

In experiments with artificially delayed data, standard PPO's performance degraded immediately with even a single iteration of staleness. By contrast, the decoupled PPO-EWMA objective maintained strong performance even when data was delayed by up to 8 iterations (over 500,000 environment steps).

Comparison of PPO staleness tolerance. The decoupled objective (center) maintains performance significantly better than standard PPO under high staleness.

Batch Size-Invariance

When applying all the proposed scaling rules, PPG-EWMA maintained nearly identical learning curves across drastically different batch sizes. Even when the batch size was scaled down by a massive factor of 256 (from 256 parallel environments down to just 1), the difference in final normalized return was a negligible 0.052.

PPG-EWMA learning curves across batch sizes. With all adjustments (left), performance is highly invariant even when batch size is reduced by a factor of 256.

Outright Performance

Beyond fixing invariance and staleness, averaging weights in parameter space acts as a variance reduction technique. In head-to-head comparisons, PPO-EWMA and PPG-EWMA slightly but consistently outperformed their standard counterparts across the Procgen environments.

Insights and Limitations

This work suggests that PPO is better viewed conceptually as approximating a natural policy gradient (bounding the speed of policy changes) rather than a strict trust region method (staying close to the specific data-collecting policy).

However, the approach does come with limitations. Maintaining the EWMA requires storing an extra copy of the network weights in memory and performing an extra forward pass during policy gradient steps, increasing the computational cost of PPO by ~30% and PPG by ~2.3%. Additionally, the mathematical rules for invariance only apply in the "small batch regime" (below the critical batch size where gradient signal-to-noise ratio is around 1), and one outlier environment ("Heist") showed unexplained performance degradation at smaller batch sizes.

Summary updated

Abstract

We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.
BibTeX
@inproceedings{hilton2022batch,
  author = {Jacob Hilton and Karl Cobbe and John Schulman},
  title = {{Batch size-invariance for policy optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=lXuZaxEaI7},
}
Figure