Introduction
Large-batch training is a critical technique for leveraging massive distributed computing clusters in deep learning. By increasing the batch size, researchers can improve hardware utilization and significantly reduce training time. However, scaling batch sizes often leads to a "generalization gap," where model accuracy degrades unless hyperparameters are meticulously hand-tuned. Furthermore, most large-batch research has focused on Convolutional Neural Networks (CNNs), leaving Recurrent Neural Networks (RNNs/LSTMs) largely unexplored in this context.
This paper introduces Linear-Epoch Gradual-Warmup (LEGW), a scheduling strategy that enables training both CNNs and LSTMs with massive batch sizes without accuracy loss. By scaling the warmup period linearly with the batch size, LEGW makes the theoretically grounded "Square Root Scaling" rule effective in practice.
The Challenge of Large-Batch Training
The primary obstacles to large-batch training are the generalization gap and architecture-specific biases. While theory suggests a Square Root Scaling rule—scaling the learning rate by $\sqrt{k}$ when the batch size increases by $k$ to maintain constant gradient variance—it often fails in practical applications. Most practitioners default to Linear Scaling, which typically breaks down once batch sizes exceed a certain threshold (e.g., 8K for ImageNet).
Linear-Epoch Gradual-Warmup (LEGW)
The proposed LEGW method is built on a simple yet effective principle: if the batch size is increased by a factor of $k$, the number of warmup epochs must also be increased by a factor of $k$.
The core components of LEGW include:
- Square Root Scaling: Adopting the theoretically preferred $\eta_{new} = \eta_{base} \times \sqrt{k}$ for the learning rate.
- Linear-Epoch Warmup: Unlike traditional gradual warmup, which uses a fixed number of epochs regardless of batch size, LEGW scales the duration. For instance, if a baseline batch size of 256 uses 5 warmup epochs, a batch size of 1024 would utilize 20 warmup epochs.
- LARS Integration: For CNN architectures, LEGW is combined with Layer-wise Adaptive Rate Scaling (LARS) to mitigate vanishing or exploding gradients in extremely large batches.
Theoretical Motivation: The Peak Shift
The authors justify LEGW through the lens of the local Lipschitz constant ($L(x, g)$), which measures the smoothness of the loss landscape. They observed that the gradient's volatility—the "peak"—occurs early in training. Crucially, as the batch size increases, this peak shifts further to the right (later in the training process).
A fixed-length warmup fails because it concludes before the model has traversed this volatile region. By scaling the warmup duration linearly with the batch size, LEGW ensures the model remains in a stable learning phase until it clears the peak volatility.
Experimental Results
The versatility of LEGW was evaluated across multiple benchmarks, including MNIST, Penn Treebank (PTB), WMT16 (GNMT), and ImageNet (ResNet-50).
LSTM and Translation Performance
On the PTB dataset, the authors successfully scaled the batch size by 64x (from 20 to 1280) with no loss in perplexity and zero manual hyperparameter tuning. For Neural Machine Translation (GNMT), scaling the batch size from 256 to 4096 reduced training time on a TPU-v2 from over 2 hours to just 33 minutes. Overall, LEGW achieved an average 5.3x speedup across four LSTM-based applications.
CNN Robustness on ImageNet
LEGW demonstrates remarkable robustness for CNNs. On ImageNet with ResNet-50, the method maintained a steady ~75.3% accuracy up to a 32K batch size, outperforming previous state-of-the-art auto-tuning techniques.
Conclusion
LEGW bridges the gap between theoretical scaling rules and practical performance. By acting as an "auto-tuner," it allows researchers to find optimal hyperparameters for small batches and seamlessly scale to massive batches. While hardware memory remains a constraint, LEGW provides a robust framework for accelerating deep learning training across diverse architectures.
Summary updated