Abstract
In deep learning, the prevailing trend is to increase batch sizes to maximize hardware utilization. However, recent research from Mila and Google DeepMind suggests that for value-based Reinforcement Learning (RL), smaller batch sizes—specifically 8 or 16—significantly outperform the industry-standard batch size of 32. This "small batch effect" enhances sample efficiency and final performance by injecting beneficial noise that aids exploration and preserves the network's plasticity, preventing the common issue of rank collapse.
Introduction
The conventional wisdom in deep learning suggests that larger batch sizes lead to more stable gradient estimates and better hardware utilization. While this holds true for supervised learning, Reinforcement Learning (RL) presents a unique set of challenges. RL agents must learn from non-stationary data distributions and use bootstrapping, where the agent learns from its own evolving predictions.
A seminal study presented at NeurIPS 2023 challenges the de facto standard of using a batch size of 32 in deep RL. By investigating the impact of mini-batch sizes on value-based agents, researchers found that reducing the batch size can lead to substantial gains in both performance and training efficiency.
The Challenges of Stability in RL
Since the introduction of the Deep Q-Network (DQN) in 2015, a batch size of 32 has been the default setting for most replay-based RL algorithms. The rationale was to provide enough stability to the gradient updates. However, the authors of this study argue that the stability provided by large batches might actually be detrimental in RL.
Unlike supervised learning, where the dataset is fixed, an RL agent's "dataset" (the replay buffer) changes as the agent interacts with the environment. This non-stationarity, combined with the feedback loops inherent in bootstrapping, means that a certain level of stochasticity or "noise" in the updates can prevent the agent from converging prematurely to suboptimal policies.
Methodology and Metrics
The researchers conducted an extensive empirical evaluation using the Dopamine library, focusing on the Arcade Learning Environment (ALE). They tested various agents, including DQN, Rainbow, QR-DQN, and IQN, across a suite of 60 Atari 2600 games.
Beyond simple performance scores, the study employed several "network health" metrics to understand why smaller batches worked:
- Effective Rank ($srank$): A measure of the dimensionality of the network's representations. A higher $srank$ indicates that the network is utilizing its capacity more effectively.
- Dormant Neurons: The percentage of neurons that have stopped contributing to the learning process. High numbers of dormant neurons indicate a loss of "plasticity," where the network becomes unable to adapt to new information.
The Small Batch Effect
The study's primary finding is the "Small Batch Effect": reducing the batch size from 32 to 8 or 16 consistently improved performance across modern RL agents.
Noise as a Feature
Smaller batches result in noisier gradient estimates. In the context of RL, this noise acts as a form of implicit regularization and exploration. It prevents the network from getting stuck in local optima and helps maintain a diverse set of features. The researchers validated this by manually adding noise to learning targets, which produced similar performance improvements.
Preserving Plasticity
A critical issue in deep RL is "rank collapse," where the network's representations become increasingly low-dimensional over time. The study found that smaller batches help maintain a higher $srank$ and reduce the number of dormant neurons. By keeping the network "plastic," small batches allow the agent to continue learning effectively throughout the entire training process.
Interaction with Multi-step Returns
Interestingly, the benefits of small batches are most significant when combined with multi-step ($n$-step) returns. This explains why the original DQN (which uses 1-step returns) did not benefit as much from smaller batches, whereas modern agents like Rainbow and QR-DQN see dramatic improvements.
Key Results and Performance
The empirical results were striking:
- QR-DQN Performance: Reducing the batch size from 32 to 8 led to a 37% improvement in aggregate human-normalized scores.
- Training Speed: Contrary to expectations that smaller batches might slow down training due to reduced parallelism, the researchers observed a 29% speedup in wall-clock time for QR-DQN. This is likely due to a smaller memory footprint allowing for better machine throughput.
- Architectural Scaling: The effect was even more pronounced when using larger architectures like ResNet. Larger models appeared to require the regularization provided by smaller batches to avoid overfitting to early, suboptimal experiences.
Conclusion and Future Directions
The findings suggest a simple yet powerful heuristic for RL practitioners: experiment with smaller batch sizes. While the study focused on value-based methods, it opens the door for similar investigations into actor-critic methods like PPO or SAC.
Future work remains to reconcile these findings with the requirements of large-scale distributed RL, where high throughput often necessitates larger batches. However, for most standard RL applications, the "small batch" approach offers a rare "free lunch"—improving both the quality of the learned agent and the speed at which it is trained.
Summary updated