Abstract
Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size $m$. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between $m = 2$ and $m = 32$, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.
Introduction
In the pursuit of reduced training times, the deep learning community has increasingly adopted large-batch training to maximize hardware utilization across distributed systems. However, this shift often introduces a "generalization gap," where models trained with large mini-batches fail to match the test performance of their small-batch counterparts. To compensate, practitioners frequently rely on heuristics such as linear learning rate scaling and gradual warm-up. This paper re-examines the fundamental dynamics of Stochastic Gradient Descent (SGD) to investigate how batch size influences training stability and final model quality when the total computational budget—defined by the number of gradient calculations—is held constant.
Methodology
The authors conducted an extensive empirical study across various architectures, including AlexNet and several ResNet variants, using the CIFAR-10, CIFAR-100, and ImageNet datasets. Their methodology centers on several key choices:
- Constant Computational Cost: Models are trained for a fixed number of epochs rather than iterations, ensuring that the total number of individual training examples processed remains identical across all batch sizes.
- Base Learning Rate ($\tilde{\eta}$): The study adopts a formalization where the effective learning rate $\eta$ is proportional to the batch size $m$, equivalent to the popular linear scaling rule.
- Stability Analysis: The research evaluates not only peak accuracy but also the "stability window"—the range of learning rates that allow a model to converge reliably without divergence.
Results and Discussion
The Optimal Batch Size and the "Stability Cliff"
The experimental results consistently demonstrate that the highest test accuracies are achieved with mini-batch sizes between $m=2$ and $m=32$. As the mini-batch size increases beyond this range, the window of stable learning rates shrinks significantly, leading to what the authors describe as a "stability cliff."
Large batches are far more sensitive to hyperparameter tuning; a learning rate that is slightly too high can cause training to become unstable. In contrast, small batches are robust across a much wider range of learning rates, providing more reliable convergence. This trend is consistent across different experimental setups.
Gradient Freshness and Update Variance
The performance advantage of small batches is largely attributed to "gradient freshness." Small batches allow for more frequent weight updates, ensuring that each gradient is calculated using the most up-to-date model parameters. Conversely, large-batch training calculates a massive average gradient based on "stale" weights that have not been updated for a long period, which can lead to suboptimal optimization trajectories.
Furthermore, the authors point out that linear scaling of the learning rate (intended to keep the average update constant) causes the variance of the weight updates to increase linearly with the batch size. This increased noise in large batches likely contributes to the observed loss of stability.
Resilience of Batch Normalization
Batch Normalization (BN) was found to be remarkably resilient to small batch sizes. Even with $m=4$, BN performs well. Interestingly, the authors discovered that using a small batch size for BN statistics (e.g., 8) while using a much larger batch size for weight updates (e.g., 1024) can actually improve performance, suggesting that the stochasticity introduced by small-batch BN may have a regularizing effect.
Conclusion
While large-batch training is currently favored for its efficiency on standard GPU hardware, this research highlights that small mini-batches (2–32) are algorithmically superior for generalization and training stability. The findings suggest that the industry's focus on massive parallelism may be at odds with optimal model performance. This work motivates a potential shift in hardware design toward processors capable of handling many small, independent tasks efficiently, allowing practitioners to leverage the benefits of small-batch training without sacrificing throughput.
Summary updated