The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning

Johan Samir Obando Ceron, Marc G Bellemare, Pablo Samuel CastroOpenReview 2023 · Submission to Tiny Papers @ ICLR · OpenReview

Abstract

In deep reinforcement learning (RL), conventional wisdom suggests that larger batch sizes reduce gradient variance and lead to more stable learning. This paper uncovers a "small batch size anomaly": when using multi-step ($n$-step) learning, reducing the batch size actually prevents performance collapse and significantly improves results while simultaneously cutting down on computation time.

Introduction

Deep Reinforcement Learning (DRL) is notoriously sensitive to hyperparameters. One of the most common techniques to improve performance is multi-step learning (or $n$-step returns), where the agent looks several steps into the future to calculate rewards rather than just one. While powerful, $n$-step learning increases the variance of updates, which can lead to "performance collapse"—a state where the agent's learning suddenly fails.

In the world of supervised learning, the standard fix for high variance is to increase the batch size. However, the RL community has largely stuck to a default batch size of 32 since the original DQN paper in 2015. The authors investigate whether this "one-size-fits-all" batch size is actually holding back modern agents that rely on multi-step updates.

Methodology

The authors conducted a systematic empirical study using the Dopamine research framework, testing four prominent value-based RL agents:

  1. DQN: The classic baseline.
  2. Rainbow: A combination of several RL improvements.
  3. QR-DQN: Quantile Regression DQN (distributional RL).
  4. IQN: Implicit Quantile Networks.

They varied the batch sizes (8, 16, 32, and 64) and the multi-step horizon ($n$) to observe how these two parameters interact. The agents were evaluated on the Arcade Learning Environment (ALE), using a diverse subset of 20 Atari 2600 games. Each agent was trained for 100 million frames across three independent seeds.

Key Insights: The $n$-step Trigger

The "anomaly" is directly tied to multi-step learning. For agents using 1-step updates, larger batch sizes still perform better, as expected. However, as soon as $n$ increases (e.g., 3-step or higher), the relationship flips, and smaller batches become superior.

Performance vs N-Step for different batch sizes

Usually, researchers try to balance variance. This paper suggests that in $n$-step RL, "doubling down" on variance—by both increasing the look-ahead ($n$) and decreasing the batch size—actually helps the agent navigate the optimization landscape better. Furthermore, the paper points out that the batch size of 32 has become a "static parameter" due to social dynamics, proving that even settled hyperparameters should be re-evaluated as algorithms evolve.

Key Results

The empirical results demonstrate massive performance gains across multiple algorithms when reducing the batch size.

Performance of DQN, Rainbow, DQN-QR, and IQN across different batch sizes
  • Massive Performance Gains: In QR-DQN, simply reducing the batch size from 32 to 8 improved performance by nearly 70% on the tested games.
  • Stability at Scale: With a batch size of 8, the Rainbow agent remained stable and high-performing even with $n$-step values as high as 9. In contrast, the default batch size of 32 caused the agent to collapse once $n$ went beyond 3.
  • Computational Efficiency: Smaller batches require less computation per update. Agents using smaller batches reached higher scores in a fraction of the wall-clock time compared to those using larger batches.

Limitations and Future Work

The study focuses exclusively on value-based agents (DQN and its descendants). It remains to be seen if this anomaly persists in policy gradient or actor-critic methods (like PPO or SAC). Additionally, while the paper empirically proves the anomaly exists, it doesn't provide a formal mathematical proof of why the extra noise from small batches prevents $n$-step collapse. Finally, testing on continuous control tasks would help determine if this is a universal property of $n$-step reinforcement learning.

Summary updated

Abstract

We present a surprising discovery: in deep reinforcement learning, decreasing the batch size during training can dramatically improve the agent's performance when combined with multi-step learning. Both reducing batch sizes and increasing the update horizon increase the variance of the gradients, so it is quite surprising that increased variance on two fronts yields improved performance. We perform a wide range of experiments to gain a better understanding of this phenomenon, which we denote variance double-down.
BibTeX
@misc{ceron2023small,
  author = {Johan Samir Obando Ceron and Marc G Bellemare and Pablo Samuel Castro},
  title = {{The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning}},
  year = {2023},
  url = {https://openreview.net/forum?id=G0heahVv5Y},
  howpublished = {OpenReview},
  note = {Submitted to Tiny Papers @ ICLR},
}
Figure