Batch size-invariance for policy optimization

Jacob Hilton, Karl Cobbe, John Schulman

Abstract

We define an algorithm as batch size-invariant if changes to the batch size can be largely compensated for by adjustments to other hyperparameters. While stochastic gradient descent (SGD) naturally exhibits this property at small batch sizes via learning rate scaling, popular policy optimization algorithms like Proximal Policy Optimization (PPO) do not. This lack of invariance stems from how these algorithms control policy updates using a reference policy that becomes increasingly "stale" as batch sizes decrease. In this work, we demonstrate how to achieve batch size-invariance by decoupling the proximal policy (used for update constraints) from the behavior policy (used for off-policy corrections). Our approach, utilizing an exponentially-weighted moving average (EWMA) of the policy network, enables stable scaling and more efficient use of stale data across diverse environments.

Introduction: The Scaling Challenge in RL

In supervised learning, practitioners rely on a well-established "rule of thumb": if you increase the batch size, you can typically increase the learning rate to maintain performance while reducing the number of training steps. This property, known as batch size-invariance, is crucial for scaling training across different hardware configurations, from single GPUs to massive clusters.

However, Reinforcement Learning (RL) has historically lacked this flexibility. Popular algorithms like PPO are notoriously sensitive to batch size. When a researcher changes the batch size to fit their hardware, they often find themselves in a "hyperparameter hell," where the agent's performance collapses unless a painstaking re-tuning process is undertaken. This sensitivity limits the portability and scalability of RL research.

The Root Cause: Overloaded Reference Policies

The core issue lies in the dual role of the "old" policy ($\pi_{\theta_{old}}$) in PPO. This reference policy is used for two distinct, and sometimes conflicting, tasks:

  1. Off-policy correction: Using importance sampling to account for the fact that data was collected by a previous version of the agent.
  2. Trust region constraint: Ensuring the current update doesn't deviate too far from a stable reference point to maintain training stability.

When the batch size is reduced, the number of gradient updates per environment step increases. This causes the current policy to drift much further away from the "old" policy within a single iteration. This increased "staleness" breaks the trust region constraint, leading to the instability shown in standard PPO implementations.

Decoupling the proximal and behavior policies allows PPO to maintain performance even as data becomes "stale" relative to the current policy. Standard PPO (left) fails as staleness increases, while the decoupled versions (middle, right) remain robust.

Methods: Decoupling and EWMA

To solve this, we introduce a fundamental architectural change: decoupling the Behavior Policy ($\pi_{\theta_{behav}}$) from the Proximal Policy ($\pi_{\theta_{prox}}$). The behavior policy is used strictly for importance sampling ratios, while the proximal policy serves as the reference point for the trust region.

We implement this using PPO-EWMA and PPG-EWMA. Instead of using the policy from the previous iteration as the reference, we maintain an Exponentially-Weighted Moving Average (EWMA) of the policy network's weights. This provides a stable, slowly-moving target for the trust region that is independent of the data collection frequency.

To achieve true invariance, we apply a specific hyperparameter "recipe" when the batch size is divided by a factor $c$:

  • Adam Step Size: Scaled by $1/\sqrt{c}$ (the square root scaling rule).
  • EWMA Decay ($\beta_{prox}$): Adjusted so the "center of mass" (the average age of the weights) remains constant in terms of environment steps.
  • Advantage Normalization: The number of iterations used to estimate mean and variance is scaled to maintain consistent statistics.

Results: Perfect Scaling on Procgen

We evaluated our approach using the Procgen Benchmark, a suite of 16 procedurally generated environments. Our experiments demonstrate that PPG-EWMA achieves a high degree of batch size-invariance. Across 15 of the 16 environments, the performance difference between the largest and smallest batch sizes was negligible, with a mean return difference of only 0.019.

PPG-EWMA achieves near-perfect batch size-invariance across various batch sizes (left), provided all necessary hyperparameter adjustments are applied. The ablation plots (a-d) show that removing any single adjustment—such as Adam step size or EWMA scaling—breaks this invariance.

Beyond scaling, the EWMA approach offers significant stability benefits. Standard PPO often becomes unstable at very small batch sizes; however, the decoupled objective maintains stability even when running on a single parallel copy of the environment. Furthermore, the algorithm handles "stale" data (data collected many iterations ago) far better than standard PPO, which is a major advantage for asynchronous training architectures.

Conclusion

By decoupling the proximal and behavior policies, we can make policy optimization algorithms batch size-invariant. While maintaining the EWMA adds a small computational overhead (roughly 3% for PPG), the benefits are substantial: practitioners can now scale their RL training across different hardware setups with confidence, knowing that their hyperparameters will remain robust. This work not only provides a practical tool for scaling but also clarifies the underlying mechanics of trust regions in modern RL.

Summary updated

Abstract

We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.
BibTeX
@inproceedings{hilton2022batch,
  author = {Jacob Hilton and Karl Cobbe and John Schulman},
  title = {{Batch size-invariance for policy optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=lXuZaxEaI7},
}

Loading related content...

Figure