To accelerate the training of deep neural networks, practitioners routinely distribute workloads across massive clusters of accelerators. In practice, this hardware parallelization requires increasing the global mini-batch size. However, simply scaling up the batch size while keeping other hyperparameters fixed typically degrades the model's final generalization accuracy or destabilizes training entirely.
To counteract this, researchers rely on the Linear Scaling Rule: if you multiply the mini-batch size by a factor of $k$, you must also multiply the learning rate by $k$. While this rule seems deceptively simple, its success relies on balancing two distinct forces: the stochastic dynamics that govern how models generalize, and the mathematical constraints of optimization speed.
Preserving the Stochastic Temperature
To understand why linear scaling works, we must look at Stochastic Gradient Descent (SGD) not just as an optimization algorithm, but as a physical system driven by noise. The randomness inherent in selecting small, random mini-batches introduces gradient noise. This noise acts as a vital regularizer, preventing the model from getting trapped in sharp local minima and instead guiding it toward broad, flat regions of the loss landscape. Flat minima are highly desirable because they consistently correlate with superior validation performance and generalization.
The effective noise level—or "stochastic temperature"—of this process is governed by the ratio of the learning rate ($\eta$) to the batch size ($B$). As demonstrated by Jastrzebski et al. (2018), the flatness of the final trained minimum is directly driven by this $\eta/B$ ratio.
Small Batch, Small LR Large Batch, Small LR Large Batch, Large LR
(Optimal Noise) (Too Deterministic) (Preserved Noise)
_ _ _ _ _ _
/ \ / \ / \ / \ / \ / \
/ \_ _/ \ / \_ _/ \ / \_ _/ \
/ \___/ \ / \___/ \ / \___/ \
[ * ] [ * ] [ * ]
Escapes sharp minima; Gets stuck in sharp, Escapes sharp minima;
finds flat regions. poorly generalizing valleys. finds flat regions.
If you scale up the batch size $B$ without adjusting the learning rate, the gradient noise drops significantly. The optimization process becomes overly deterministic, causing the model to settle into the nearest sharp minimum, which generalizes poorly. From a Bayesian perspective, Le et al. (2018) show that the optimal batch size scales linearly with both the learning rate and the total dataset size to maintain a stable noise trajectory. Adjusting the learning rate proportionally to the batch size keeps the stochastic temperature invariant, ensuring the model navigates the loss landscape identically regardless of the hardware configuration.
Invariance of Convergence Rates Per Epoch
Beyond generalization, linear scaling satisfies a fundamental optimization requirement: maintaining consistent progress per training epoch.
When you increase the mini-batch size, the model processes more data points per iteration, which means it performs fewer total weight updates per epoch. If the step size remains unchanged, the model will travel a much shorter distance across the loss landscape over the course of an epoch, severely slowing down convergence. To ensure that the model makes the same amount of progress per epoch, each individual step must be larger.
Nacson et al. (2019) provided a rigorous theoretical foundation for this mechanism. They proved that when optimizing over separable data, the maximum stable learning rate is directly proportional to the mini-batch size. By scaling the learning rate linearly, the asymptotic convergence rate per epoch remains entirely invariant, allowing for seamless parallelization.
This theoretical insight explains early empirical breakthroughs across various domains. For instance, Jiang et al. (2015) discovered that ReLU-based architectures could tolerate massive mini-batches without sacrificing accuracy, provided the initial learning rate was scaled proportionally to compensate for the reduced update frequency. Similarly, Chen et al. (2014) utilized this direct correlation between parallel data streams and learning rates to achieve rapid, stable convergence when training recurrent neural network language models on highly parallelized GPU architectures.
Application in Distributed and Dynamic Training
In modern distributed deep learning, linear scaling serves as the operational backbone for training large models efficiently. In distributed frameworks like FireCaffe, developed by Iandola et al. (2016), scaling the learning rate in tandem with the global batch size allows clusters of over a hundred GPUs to maintain baseline accuracy while drastically reducing communication overhead.
The utility of this rule extends beyond static hyperparameter tuning; it also enables optimization strategies where batch sizes fluctuate dynamically during a single training run:
- Multigrid Training: In computer vision, the authors of Wu et al. (2020) introduced a multigrid method for video models that alters data resolutions during training. Because changing the resolution alters the number of samples that can fit into GPU memory, the mini-batch size fluctuates. By dynamically adjusting the learning rate in proportion to these shifting batch sizes, the authors kept the magnitude of gradient updates consistent.
- Predictable Scheduling: For large language models, Shen et al. (2024) leveraged this predictable linear relationship to design a batch-size and token-number agnostic "Power Scheduler." This approach eliminates the need for exhaustive, expensive hyperparameter sweeps when migrating workloads across different hardware clusters.
The Boundaries of Linear Scaling
Linear scaling is highly effective, but it cannot be applied indefinitely. As batch sizes grow into tens of thousands of samples, the required learning rate becomes massive. At the very beginning of training, when the model's weights are randomly initialized, taking such enormous steps can cause immediate gradient explosion and divergence.
To safely scale to these extreme regimes, practitioners combine linear scaling with specialized stabilization techniques. Lin et al. (2023) demonstrate that pairing the linear scaling rule with a temporal warmup phase—gradually increasing the learning rate from zero to its target value over the first few epochs—allows video models to exploit massive batch sizes without destabilizing early training. For even larger batches, layer-wise adaptive scaling algorithms like LARS, as detailed in a survey by Chahal et al. (2020), are required to adjust learning rates independently for each layer based on weight magnitudes.
Furthermore, certain architectures and geometries present unique challenges to strict linear scaling:
- Transformer Sensitivity: As observed by Aji et al. (2019), Transformer architectures exhibit extreme sensitivity to the high learning rates demanded by large batch sizes, often requiring sublinear scaling strategies to prevent convergence degradation.
- Non-Euclidean Optimization: When optimizing over non-Euclidean spaces, such as the Riemannian manifolds analyzed by Sakai et al. (2024), increasing the batch size reduces step complexity up to a strict "critical batch size," beyond which the computational returns of further scaling diminish.
Key Foundational Work
Rectified linear neural networks with tied-scalar regularization for LVCSR (2015) — contributed 2 concept elements to this direction.
Efficient GPU-based training of recurrent neural network language models using spliced sentence bunch (2014) — contributed 2 concept elements to this direction.