Measuring the Effects of Data Parallelism on Neural Network Training

Christopher J. Shallue, Jaehoon Lee, Joseph M. Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl

TL;DR

By systematically tuning metaparameters across 168,000+ training runs, the authors reveal a universal three-regime relationship between batch size and training speed: initial linear speedup, followed by diminishing returns, and finally a hard saturation limit. They find no inherent generalization degradation at large batch sizes, showing that prior reports of a "generalization gap" largely stemmed from suboptimal tuning heuristics, fixed epoch budgets, or batch normalization artifacts.

What Problem Does This Paper Address?

As hardware accelerators (like GPUs and TPU pods) scale, the most straightforward way to accelerate neural network training via synchronous data parallelism is to increase the mini-batch size. However, the machine learning community has long debated whether large batches inherently hurt generalization (converging to "sharp minima") and how training speed actually scales with batch size.

Much of the prior empirical literature presented conflicting claims—often because researchers tuned hyperparameters (like learning rate) only at small batch sizes and relied on rigid heuristics (e.g., linear or square-root scaling), evaluated models under restrictive epoch budgets, or confounded batch size with batch normalization statistics. This paper provides an exhaustive, rigorously tuned empirical investigation into how batch size impacts both training speed (steps to reach a target error) and out-of-sample solution quality across diverse workloads.

Methods

The authors isolate the true effect of data parallelism by measuring a hardware-agnostic metric: steps to result (the number of optimizer update steps required to reach a pre-specified out-of-sample error goal).

Key methodological decisions include:

  • Independent Metaparameter Tuning: For every single batch size and workload, hyperparameters (initial learning rate, momentum, and linear learning rate decay schedules) were independently tuned using quasi-random search with equal budgets (typically 100 non-divergent trials per batch size).
  • Decoupled Batch Normalization: To prevent changing the underlying mathematical objective when scaling batch sizes, models using Batch Normalization (such as ResNet-50) used "ghost batch normalization" with a fixed virtual batch size of 32.
  • Regularization Control: Label smoothing was evaluated to investigate whether small-batch gradient noise functions primarily as implicit regularization that can be substituted with explicit regularizers.

Datasets

The study evaluated seven vision and language datasets spanning orders of magnitude in size:

  • MNIST & Fashion MNIST: 55,000 training images; evaluated on classification error.
  • CIFAR-10: 45,000 training images; evaluated on classification error.
  • ImageNet: ~1.28 million training images; evaluated on top-1 validation error.
  • Open Images v4: ~4.5 million training images; evaluated on class-agnostic Average Precision (AP).
  • LM1B (One Billion Word Benchmark): ~30.3 million sentences; evaluated on per-token cross-entropy.
  • Common Crawl: ~25.8 billion sentences filtered web crawl data; evaluated on per-token cross-entropy.

Key Insights and Ideas

  • Universal Three-Regime Scaling: Across all models, optimizers, and datasets, the relationship between batch size and training steps follows an identical characteristic shape:
  • Perfect Linear Scaling: Doubling the batch size halves the required update steps ($b$-fold speedup).
  • Diminishing Returns: Increasing batch size yields sublinear reductions in steps.
  • Maximal Data Parallelism: A plateau where adding more examples per batch yields zero reduction in optimization steps.
  • The "Generalization Gap" is Largely an Artifact: Larger batch sizes do not inherently degrade out-of-sample test error. Prior degradation claims were largely caused by evaluating under a fixed epoch budget (which starves large batches of gradient update steps), applying failing learning rate rules of thumb, or failing to compensate for the lost regularizing effect of mini-batch noise (which label smoothing can recover).
  • Momentum Drastically Extends Scalability: At very small batch sizes, momentum provides zero step-reduction benefit over plain SGD. However, momentum dramatically extends the linear scaling regime to much larger batch sizes.
  • Common Scaling Heuristics Break Down: The popular rule of thumb—scaling the learning rate linearly with batch size—frequently fails at larger batch sizes. In many workloads, optimal effective learning rates plateau or require increasing momentum rather than the raw learning rate.
  • Model and Optimizer Matter More Than Dataset Size: The maximum useful batch size is heavily governed by the model architecture and optimizer. Merely increasing dataset size does not reliably extend the batch size scaling limits.

Key Results

  • Dataset Release: The authors released raw data from 168,160 individual model training runs consisting of 71,638,836 loss measurements across 35 distinct workloads.
  • Scaling Ceilings: The perfect scaling limit varied wildly—from batch size $2^4$ (16) up to $2^{13}$ (8,192). The maximum useful batch size plateaued between $2^9$ (512) and $2^{16}$ (65,536).
  • Momentum Advantage: On CIFAR-10 (ResNet-8), plain SGD stopped scaling linearly past a batch size of 16, whereas Nesterov momentum maintained linear scaling up to a batch size of 256.
  • Regularization Effects: For ResNet-50 on ImageNet at batch size $2^{16}$ (65,536), label smoothing recovered up to 1.0% in validation error, enabling the model to hit the target error rate (0.25) where unregularized large-batch runs failed.

Limitations and Future Work

  • Search Space Bias: Even with automated quasi-random search, search space boundaries were chosen with human judgment; globally optimal hyperparameter settings cannot be guaranteed.
  • Idealized Parallel Hardware: The study assumes an ideal system where per-step wall-clock time is invariant to batch size. Real distributed clusters incur communication and synchronization overheads that shift the practical wall-clock optimum.
  • Predicting Limits A Priori: The work empirically characterizes scaling retrospectively, but does not provide an analytical method to predict a workload's critical batch size without expensive tuning sweeps.
  • Unexplored Optimizers: The study focused on SGD and momentum variants. Evaluating second-order or curvature-aware methods (such as K-FAC or adaptive optimizers like Adam across all workloads) remains an open direction for extending parallel scaling limits even further.

Summary updated

Abstract

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.
BibTeX
@article{shallue2019measuring,
  author = {Christopher J. Shallue and Jaehoon Lee and Joseph M. Antognini and Jascha Sohl-Dickstein and Roy Frostig and George E. Dahl},
  title = {{Measuring the Effects of Data Parallelism on Neural Network Training}},
  journal = {J. Mach. Learn. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=XSx4caacGWM},
}

Loading related content...

Figure