To speed up deep learning, the most common strategy is to parallelize training across multiple GPUs. This naturally leads to using larger mini-batches. However, simply increasing the batch size often leads to a "generalization gap" where the model performs worse on validation data.
The Linear Scaling Rule is the primary heuristic used to fix this: if you increase the mini-batch size by a factor of $k$, you should also increase the learning rate by that same factor $k$.
The Intuition of Constant Update Magnitude
The most straightforward way to understand why we scale the learning rate is to look at the total change in weights over a fixed amount of data (e.g., one epoch).
In standard SGD, if we have a small batch size $B$ and a learning rate $\eta$, we perform $k$ separate updates to reach $k \times B$ samples. If we instead use a single large batch of size $k \times B$, we only perform one update. As Priya Goyal and colleagues argue in Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, to make that single large-batch update move the weights roughly as far as the $k$ small-batch updates would have, the learning rate must be $k$ times larger.
Hui Jiang and Si Wei observed this effect in Rectified linear neural networks with tied-scalar regularization for LVCSR, noting that linear scaling compensates for the reduced frequency of weight updates. By scaling the learning rate, the magnitude of parameter changes remains consistent regardless of how the data is partitioned into batches.
Maintaining the Noise Ratio
A more nuanced perspective views SGD not just as an optimizer, but as a stochastic process with a specific "noise" level. The stochasticity of SGD—the fact that gradients are estimated from random subsets of data—is actually a feature, not a bug; it helps the model escape sharp local minima and find "flatter" regions of the loss landscape that generalize better.
In Finding Flatter Minima with SGD, Stanislaw Jastrzebski and Yoshua Bengio demonstrate that the "flatness" of the final minimum is driven by the ratio of the learning rate to the batch size ($\eta/S$).
- Larger batches reduce gradient variance (less noise).
- Higher learning rates increase the step size (more noise/volatility).
By keeping the ratio $\eta/S$ constant, we preserve the "effective noise" of the training process. This ensures that the optimizer explores the landscape with the same level of aggression, eventually settling into regions with similar geometric properties. This concept is central to modern LLM training; for instance, Yikang Shen and Matthew Stallone utilize this linear relationship in their Power Scheduler to allow for seamless scaling across different hardware configurations without re-tuning hyperparameters.
Theoretical Bounds and Stability
The rule isn't just a heuristic; it has roots in convergence theory. Mor Shpigel Nacson and Daniel Soudry proved in Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate that for classification tasks on separable data, the maximum permissible learning rate for stable convergence scales linearly with the minibatch size. This allows practitioners to increase the batch size for parallelization without altering the fundamental convergence dynamics.
Similarly, adaptive methods like AdaBatch by Michael Garland and Aditya Devarakonda use this ratio to dynamically adjust batch sizes during training. By starting with small batches (high noise) and scaling up (low noise), they capture the early-stage benefits of small-batch exploration while leveraging GPU efficiency later in training.
Where the Rule Breaks Down
Linear scaling is not a universal law. There is a "critical batch size" beyond which the rule fails. Noah Golmant and Amir Gholami point out in On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent that eventually, the gradient becomes so accurate that further increasing the batch size provides no additional information.
At extreme scales—such as the 1.28 million sample batches explored by Yang You and James Demmel in The Limit of the Batch Size—the linear scaling heuristic begins to collapse. In these "huge-batch" regimes, the optimization dynamics change so fundamentally that simple proportional adjustments are no longer sufficient to maintain accuracy.