Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model

Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, Roger B. Grosse

Introduction

In the quest to train massive neural networks faster, the most common strategy is data parallelism: increasing the batch size to utilize more GPUs. However, this scaling does not continue infinitely. Every optimization problem has a critical batch size—a point beyond which doubling the batch size yields diminishing returns and no longer halves the number of training steps.

Before this research, understanding why this limit exists and how different optimizers (like SGD vs. Adam) affect it required prohibitively expensive empirical sweeps. Theoretical models were often too abstract or restricted to plain SGD. This paper introduces a "middle ground": the Noisy Quadratic Model (NQM). The authors demonstrate that while momentum only helps at large batch sizes, preconditioning techniques like Adam and K-FAC significantly extend the range of "perfect scaling," allowing researchers to use much larger batches to speed up training.

The Noisy Quadratic Model (NQM)

The researchers utilize the NQM to simulate optimization dynamics. In this setup, the loss function is treated as a convex quadratic (a bowl shape), but the optimizer only sees "noisy" versions of the gradient.

Because the model is quadratic, the dynamics of various optimizers—including SGD, Momentum, Adam, and K-FAC—can be solved or simulated analytically in seconds rather than weeks. The authors represent optimizers like Adam and K-FAC as preconditioned gradient descent, where the gradient is scaled by a matrix related to the curvature of the loss surface (the Hessian). To ensure these theoretical findings hold in the "wild," the authors validated their NQM predictions using massive experiments on real neural networks across standard benchmarks (MNIST, CIFAR-10, and the LM1B Transformer dataset).

Key Insights and Scaling Behaviors

The Convergence vs. Noise Trade-off

Every step in stochastic optimization involves a tug-of-war. Higher learning rates help the model converge faster to the minimum, but they also increase the "steady-state risk"—the noise floor that prevents reaching the exact bottom. Larger batch sizes reduce this noise, allowing for higher learning rates and faster convergence. The study confirmed that the optimal learning rate scales linearly with batch size up until the critical batch size is reached, after which it saturates.

Momentum is a "Large Batch" Specialist

The NQM predicts, and experiments confirm, that momentum provides almost no benefit at small batch sizes where noise dominates. It only shines at larger batch sizes where the gradient estimates are stable enough to benefit from acceleration.

Preconditioning Extends the Horizon

Techniques like Adam and K-FAC fundamentally change the "shape" of the optimization problem. This allows for "perfect scaling" (linear speedup) to continue to much larger batch sizes than is possible with SGD or Momentum.

Scaling behavior of Momentum and Preconditioning

In terms of scaling hierarchy, the results consistently showed K-FAC > Adam > Momentum SGD > Plain SGD. K-FAC demonstrated the largest critical batch sizes, making it the most "parallelizable" optimizer tested. On the LM1B dataset, preconditioned optimizers showed a massive advantage, pushing the limits of parallel training significantly further than momentum-based methods.

Exponential Moving Averaging (EMA)

The authors also found that Exponential Moving Averaging (EMA) of the weights helps lower the noise floor. The NQM suggests that EMA can achieve the same accuracy as a larger batch size but with fewer total computations, acting as a computational saver.

Limitations and Future Work

The primary limitation of this framework is the quadratic assumption. Real neural network landscapes are highly non-convex and contain plateaus, saddles, and cliffs. While the NQM works remarkably well because local regions of the loss surface often behave quadratically, it may fail to capture phenomena related to escaping local minima or complex learning rate schedules (like restarts).

Future work could involve using the NQM to automatically tune hyperparameters or to design new optimization algorithms specifically tailored to exploit massive parallel hardware, pushing the boundaries of data parallelism even further.

Summary updated

Abstract

Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, through two different lenses: large scale experiments and analysis using a simple noisy quadratic model (NQM). We experimentally demonstrate that optimization algorithms that employ preconditioning, specifically Adam and K-FAC, result in much larger critical batch sizes than stochastic gradient descent with momentum. We also demonstrate that the NQM captures many of the essential features of real neural network training, despite being drastically simpler to work with. The NQM predicts our results with preconditioned optimizers, previous results with accelerated gradient descent, and other results around optimal learning rates and large batch training, making it a useful tool to generate testable predictions about neural network optimization. We demonstrate empirically that the simple noisy quadratic model (NQM) displays many similarities to neural networks in terms of large-batch training. We prove analytical convergence results for the NQM model that predict such behavior and hence provide possible explanations and a better understanding for many large-batch training phenomena.
BibTeX
@inproceedings{zhang2019which,
  author = {Guodong Zhang and Lala Li and Zachary Nado and James Martens and Sushant Sachdeva and George E. Dahl and Christopher J. Shallue and Roger B. Grosse},
  title = {{Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=26ed3q-Dvq6},
}
Figure