Measuring the Effects of Data Parallelism on Neural Network Training

Christopher J. Shallue, Jaehoon Lee, Joseph M. Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl

Abstract

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

Introduction

As hardware accelerators like GPUs and TPUs become more powerful, the most straightforward way to utilize their massive parallel processing power is to increase the batch size—the number of training examples processed before updating the model's internal weights.

However, the deep learning community has been divided on the effects of "data parallelism." Some researchers argued that large batches lead to "sharp minima" and poor generalization, while others suggested a "critical batch size" beyond which parallelism provides no benefit. This paper presents a massive empirical study—encompassing over 160,000 trained models—to provide a definitive characterization of these effects.

Methodology and Datasets

The researchers adopted a rigorous "steps-to-result" methodology. Instead of training for a fixed number of epochs, they defined a goal (e.g., a specific validation error) and measured how many training steps were required to reach it across different batch sizes. A key differentiator was independent metaparameter tuning: for every batch size and workload, they used quasi-random search to optimize the learning rate, momentum, and learning rate schedules. This ensured they were comparing the "best possible" version of a model at each scale.

The study covered seven diverse datasets across image and text domains, including ImageNet, Open Images, and massive subsets of Common Crawl, providing a wide range of scales to test if scaling laws held across different orders of magnitude.

The Universal Scaling Law and Momentum

Across every workload, the relationship between batch size and training steps follows a predictable three-stage curve: perfect linear scaling, followed by diminishing returns, and finally a "maximal data parallelism" wall where increasing batch size provides zero reduction in steps.

One of the most striking findings was that optimizers with momentum allow models to scale to much larger batch sizes than plain SGD. Momentum effectively pushes the wall of diminishing returns further out.

Comparison of update rules for SGD with momentum and Nesterov momentum.

Key Results

  • The "Generalization Gap" is a Myth: The authors found no evidence that large batches necessarily hurt model quality. Previous reports of degradation were often due to insufficient tuning or comparing models under unfair "epoch budgets."
  • Massive Variation in Scaling: The "critical batch size" varied wildly. For some simple models, it was as low as 16, while for ResNet-50 on ImageNet or Transformers on LM1B, perfect scaling continued up to 8,192 or even 65,536.
  • Heuristics are Unreliable: Popular rules of thumb, such as linearly scaling the learning rate with the batch size, do not hold up at large scales; the optimal relationship is complex and workload-dependent.

Limitations and Future Work

While the paper describes scaling behavior retrospectively, it does not provide a way to predict the critical batch size for a new workload without expensive experiments. Future work is needed to see if adaptive methods like Adam or second-order optimizers can push the limits of data parallelism even further. Additionally, the authors noted that while large batches don't inherently hurt generalization, they might require different regularization techniques to match the implicit regularization provided by the noise in small-batch training.

Summary updated

Abstract

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.
BibTeX
@article{shallue2019measuring,
  author = {Christopher J. Shallue and Jaehoon Lee and Joseph M. Antognini and Jascha Sohl-Dickstein and Roy Frostig and George E. Dahl},
  title = {{Measuring the Effects of Data Parallelism on Neural Network Training}},
  journal = {J. Mach. Learn. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=XSx4caacGWM},
}
Figure