Abstract
In deep reinforcement learning (RL), conventional wisdom suggests that larger batch sizes reduce gradient variance and lead to more stable learning. This paper uncovers a "small batch size anomaly": when using multi-step ($n$-step) learning, reducing the batch size actually prevents performance collapse and significantly improves results while simultaneously cutting down on computation time.
Introduction
Deep Reinforcement Learning (DRL) is notoriously sensitive to hyperparameters. One of the most common techniques to improve performance is multi-step learning (or $n$-step returns), where the agent looks several steps into the future to calculate rewards rather than just one. While powerful, $n$-step learning increases the variance of updates, which can lead to "performance collapse"—a state where the agent's learning suddenly fails.
In the world of supervised learning, the standard fix for high variance is to increase the batch size. However, the RL community has largely stuck to a default batch size of 32 since the original DQN paper in 2015. The authors investigate whether this "one-size-fits-all" batch size is actually holding back modern agents that rely on multi-step updates.
Methodology
The authors conducted a systematic empirical study using the Dopamine research framework, testing four prominent value-based RL agents:
- DQN: The classic baseline.
- Rainbow: A combination of several RL improvements.
- QR-DQN: Quantile Regression DQN (distributional RL).
- IQN: Implicit Quantile Networks.
They varied the batch sizes (8, 16, 32, and 64) and the multi-step horizon ($n$) to observe how these two parameters interact. The agents were evaluated on the Arcade Learning Environment (ALE), using a diverse subset of 20 Atari 2600 games. Each agent was trained for 100 million frames across three independent seeds.
Key Insights: The $n$-step Trigger
The "anomaly" is directly tied to multi-step learning. For agents using 1-step updates, larger batch sizes still perform better, as expected. However, as soon as $n$ increases (e.g., 3-step or higher), the relationship flips, and smaller batches become superior.
Usually, researchers try to balance variance. This paper suggests that in $n$-step RL, "doubling down" on variance—by both increasing the look-ahead ($n$) and decreasing the batch size—actually helps the agent navigate the optimization landscape better. Furthermore, the paper points out that the batch size of 32 has become a "static parameter" due to social dynamics, proving that even settled hyperparameters should be re-evaluated as algorithms evolve.
Key Results
The empirical results demonstrate massive performance gains across multiple algorithms when reducing the batch size.
- Massive Performance Gains: In QR-DQN, simply reducing the batch size from 32 to 8 improved performance by nearly 70% on the tested games.
- Stability at Scale: With a batch size of 8, the Rainbow agent remained stable and high-performing even with $n$-step values as high as 9. In contrast, the default batch size of 32 caused the agent to collapse once $n$ went beyond 3.
- Computational Efficiency: Smaller batches require less computation per update. Agents using smaller batches reached higher scores in a fraction of the wall-clock time compared to those using larger batches.
Limitations and Future Work
The study focuses exclusively on value-based agents (DQN and its descendants). It remains to be seen if this anomaly persists in policy gradient or actor-critic methods (like PPO or SAC). Additionally, while the paper empirically proves the anomaly exists, it doesn't provide a formal mathematical proof of why the extra noise from small batches prevents $n$-step collapse. Finally, testing on continuous control tasks would help determine if this is a universal property of $n$-step reinforcement learning.
Summary updated