In the world of deep learning, there is a piece of "folk wisdom" that almost every practitioner follows: start your training with a large learning rate and drop it (anneal it) later. If you start with a small learning rate, your training loss might drop faster initially, but your final model will almost certainly generalize poorly.
While the community has known that this happens for years, we haven't fully understood why. A paper from NeurIPS 2019 by Yuanzhi Li, Colin Wei, and Tengyu Ma titled "Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks" provides a compelling answer centered on a concept they call Learning Order.
TL;DR
The paper argues that a large initial learning rate acts as a filter: the resulting Stochastic Gradient Descent (SGD) noise prevents the model from memorizing "easy-to-fit but noisy" patterns early on. This forces the network to prioritize learning "hard-to-fit but robust" features first, leading to significantly better generalization after the learning rate is eventually reduced.
What Problem Does This Paper Address?
Deep learning training curves often show a paradox. A model trained with a small learning rate frequently outperforms a large-learning-rate model in both training and test error during the early epochs. However, once the large learning rate is annealed (reduced), its accuracy spikes, eventually surpassing the small-learning-rate model.
Existing explanations often point to the "sharpness" of local minima—suggesting large learning rates help the model find "flatter" regions of the loss landscape. This paper argues that sharpness isn't the whole story. Instead, they focus on the dynamics of training: the specific order in which a network chooses to learn different types of patterns in the data.
Methods
The authors approach the problem through both theoretical proofs and empirical experiments:
- Theoretical Framework: They analyze a two-layer ReLU network trained with SGD and spherical Gaussian noise. They construct a specific data distribution containing two types of patterns:
- Pattern $\mathcal{P}$: Hard-to-fit (requires many steps) but easy-to-generalize (robust features).
- Pattern $\mathcal{Q}$: Easy-to-fit (can be memorized quickly) but hard-to-generalize (noisy or spurious shortcuts).
- Algorithm Comparison: They compare "Algorithm L-S" (Large then Small LR) against "Algorithm S" (Small LR only).
- Noise Injection: Based on their theory that noise is the key regularizer, they propose a mitigation strategy for small-LR training: injecting Gaussian noise into the network's activations (pre-activation noise) and annealing that noise over time.
Datasets
- Synthetic Data: A custom-designed distribution used for the mathematical proofs, where labels are determined by a mix of the $\mathcal{P}$ and $\mathcal{Q}$ patterns.
- CIFAR-10: Used for standard validation.
- Patch-Augmented CIFAR-10: The authors created a modified version of CIFAR-10 where a small, memorizable $7 \times 7$ pixel patch was added to a subset of images. This patch acts as a "shortcut" (an easy-to-fit, non-generalizing pattern) to test if different learning rates would prioritize it differently.
Key Insights and Ideas
- The "Easy-to-Fit" Trap: Small learning rates have very little SGD noise. This allows the model to immediately "see" and memorize simple, noisy patterns (like the artificial patch). Once the model fits these shortcuts, the gradients for the more complex, robust features diminish, and the model never learns the "real" data properly.
- Noise as a Filter: A large learning rate creates high SGD noise. This noise effectively "drowns out" the signal from easy-to-memorize patterns. The model is forced to ignore the "easy" shortcuts because they are too small to be seen through the noise. It can only make progress on patterns that provide a strong, consistent gradient signal—the robust features.
- Learning Order Matters: Generalization isn't just about where you end up in the loss landscape; it's about the path you took to get there. By forcing the model to learn robust features first, the large learning rate sets a foundation that prevents overfitting later.
Key Results
- Theoretical Gap: The authors prove that for their constructed dataset, the Large-Small LR schedule achieves a test error of $O(p\kappa)$ (where $\kappa$ is a small value), while the Small LR model is mathematically stuck with a much higher error of at least $\Omega(p)$.
- The Patch Experiment: On the patch-augmented CIFAR-10, the small-LR model's accuracy on the "patch-only" images skyrocketed to nearly 100% almost immediately, while its accuracy on clean images suffered (70%). In contrast, the large-LR model ignored the patch entirely until the annealing step, eventually achieving much higher accuracy on clean images (83%).
- Noise as a Substitute: By adding pre-activation noise to a model trained with a small learning rate, the authors were able to mimic the benefits of a large LR, seeing a 4.72% increase in validation accuracy on CIFAR-10.
Limitations and Future Work
- Model Complexity: The theoretical proofs are limited to two-layer networks. While the intuition carries over to deep ResNets in their experiments, a formal proof for deep architectures remains an open challenge.
- Noise Simplification: The paper uses spherical Gaussian noise as a proxy for the complex noise generated by mini-batch SGD. Real-world SGD noise is anisotropic (directional), which might have additional regularizing properties not captured here.
- Future Directions: The authors suggest that designing better algorithms to explicitly control "learning order" (perhaps through noise injection or architectural choices) could be a fruitful path for developing more robust training techniques.
Summary updated