Abstract
Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes easy-to-generalize, hard-to-fit patterns, it generalizes worse on hard-to-generalize, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.
Introduction
In deep learning, a common heuristic is to start training with a large learning rate (LR) and reduce it later. While this often results in slower initial progress compared to starting with a small LR, it consistently leads to superior final generalization. This paper investigates the underlying dynamics of this phenomenon, proposing that the initial learning rate dictates the order in which a network learns different data patterns.
Practitioners have long noted a paradox: models trained with a small LR from the start often achieve lower training loss and higher test accuracy in the early stages of training. However, once the learning rate of a model that started large is annealed (reduced), its performance quickly leapfrogs that of the small-LR model. This research moves beyond static explanations like local minima sharpness to explore how learning rate choice filters the features a model prioritizes.
The Learning Order Hypothesis
The core of the authors' argument is the "Learning Order" hypothesis. They posit that the learning rate acts as a filter for pattern complexity:
- Hard-to-fit, easy-to-generalize patterns ($\mathcal{P}$): These are robust features (like linear separability) that require many samples to learn.
- Easy-to-fit, hard-to-generalize patterns ($\mathcal{Q}$): These are complex, non-linear patterns or noise that can be memorized quickly with few samples but do not generalize well.
A large initial LR creates significant stochastic noise, preventing the model from latching onto the complex, low-noise $\mathcal{Q}$ patterns early on. This forces the network to focus on the simpler, more robust $\mathcal{P}$ features. Conversely, a small LR is "too efficient," allowing the model to prematurely memorize spurious correlations before it has captured the underlying data structure. Once the model "solves" the training set via these shortcuts, it stops learning the more useful, harder features.
Experimental Results
The authors validated their hypothesis through both theoretical proofs and empirical tests on CIFAR-10.
CIFAR-10 Dynamics
In standard CIFAR-10 training, the small-LR model reaches high training accuracy faster but plateaus at a lower validation accuracy. As shown in the figures below, the large-LR model experiences a dramatic performance jump immediately following the annealing step, eventually surpassing the small-LR model.
The Patch Experiment
To further isolate this effect, the authors used a "Patch-Augmented CIFAR-10" dataset. They added a $7 \times 7$ pixel patch—an easily memorizable but spurious feature—to a subset of images. The small-LR model immediately memorized the patch, leading to poor performance on clean images (69.89%). In contrast, the large-LR model ignored the patch until after annealing, allowing it to learn the actual image features first and achieve 87.61% accuracy on clean images.
Noise as a Regularizer
The research also demonstrates that the regularization effect of a large LR can be mimicked by injecting Gaussian noise into pre-activations during small-LR training. On CIFAR-10, adding this noise improved a small-LR model's accuracy from 84.93% to 89.65%, nearly matching the 90.41% achieved by the standard large-LR schedule.
Conclusion
This work provides a compelling explanation for the effectiveness of large initial learning rates: they act as a temporal regularizer that enforces a beneficial learning order. By preventing the early memorization of noisy, easy-to-fit patterns, a large LR ensures the model prioritizes robust features that lead to better generalization. Future work may involve developing automated noise-injection techniques to achieve these benefits without the need for manual learning rate schedule tuning.
Summary updated