Abstract
We define an algorithm as batch size-invariant if changes to the batch size can be largely compensated for by adjustments to other hyperparameters. While stochastic gradient descent (SGD) naturally exhibits this property at small batch sizes via learning rate scaling, popular policy optimization algorithms like Proximal Policy Optimization (PPO) do not. This lack of invariance stems from how these algorithms control policy updates using a reference policy that becomes increasingly "stale" as batch sizes decrease. In this work, we demonstrate how to achieve batch size-invariance by decoupling the proximal policy (used for update constraints) from the behavior policy (used for off-policy corrections). Our approach, utilizing an exponentially-weighted moving average (EWMA) of the policy network, enables stable scaling and more efficient use of stale data across diverse environments.
Introduction: The Scaling Challenge in RL
In supervised learning, practitioners rely on a well-established "rule of thumb": if you increase the batch size, you can typically increase the learning rate to maintain performance while reducing the number of training steps. This property, known as batch size-invariance, is crucial for scaling training across different hardware configurations, from single GPUs to massive clusters.
However, Reinforcement Learning (RL) has historically lacked this flexibility. Popular algorithms like PPO are notoriously sensitive to batch size. When a researcher changes the batch size to fit their hardware, they often find themselves in a "hyperparameter hell," where the agent's performance collapses unless a painstaking re-tuning process is undertaken. This sensitivity limits the portability and scalability of RL research.
The Root Cause: Overloaded Reference Policies
The core issue lies in the dual role of the "old" policy ($\pi_{\theta_{old}}$) in PPO. This reference policy is used for two distinct, and sometimes conflicting, tasks:
- Off-policy correction: Using importance sampling to account for the fact that data was collected by a previous version of the agent.
- Trust region constraint: Ensuring the current update doesn't deviate too far from a stable reference point to maintain training stability.
When the batch size is reduced, the number of gradient updates per environment step increases. This causes the current policy to drift much further away from the "old" policy within a single iteration. This increased "staleness" breaks the trust region constraint, leading to the instability shown in standard PPO implementations.
Methods: Decoupling and EWMA
To solve this, we introduce a fundamental architectural change: decoupling the Behavior Policy ($\pi_{\theta_{behav}}$) from the Proximal Policy ($\pi_{\theta_{prox}}$). The behavior policy is used strictly for importance sampling ratios, while the proximal policy serves as the reference point for the trust region.
We implement this using PPO-EWMA and PPG-EWMA. Instead of using the policy from the previous iteration as the reference, we maintain an Exponentially-Weighted Moving Average (EWMA) of the policy network's weights. This provides a stable, slowly-moving target for the trust region that is independent of the data collection frequency.
To achieve true invariance, we apply a specific hyperparameter "recipe" when the batch size is divided by a factor $c$:
- Adam Step Size: Scaled by $1/\sqrt{c}$ (the square root scaling rule).
- EWMA Decay ($\beta_{prox}$): Adjusted so the "center of mass" (the average age of the weights) remains constant in terms of environment steps.
- Advantage Normalization: The number of iterations used to estimate mean and variance is scaled to maintain consistent statistics.
Results: Perfect Scaling on Procgen
We evaluated our approach using the Procgen Benchmark, a suite of 16 procedurally generated environments. Our experiments demonstrate that PPG-EWMA achieves a high degree of batch size-invariance. Across 15 of the 16 environments, the performance difference between the largest and smallest batch sizes was negligible, with a mean return difference of only 0.019.
Beyond scaling, the EWMA approach offers significant stability benefits. Standard PPO often becomes unstable at very small batch sizes; however, the decoupled objective maintains stability even when running on a single parallel copy of the environment. Furthermore, the algorithm handles "stale" data (data collected many iterations ago) far better than standard PPO, which is a major advantage for asynchronous training architectures.
Conclusion
By decoupling the proximal and behavior policies, we can make policy optimization algorithms batch size-invariant. While maintaining the EWMA adds a small computational overhead (roughly 3% for PPG), the benefits are substantial: practitioners can now scale their RL training across different hardware setups with confidence, knowing that their hyperparameters will remain robust. This work not only provides a practical tool for scaling but also clarifies the underlying mechanics of trust regions in modern RL.
Summary updated