Learning Stages: Phenomenon, Root Cause, Mechanism Hypothesis, and Implications

Kaichao You, Mingsheng Long, Michael I. Jordan, Jianmin Wang

Abstract

Learning rate decay (lrDecay) is a de facto technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

Introduction

Learning rate decay (lrDecay) is a cornerstone of training modern neural networks. Despite its ubiquity, our theoretical understanding of why it works has remained surprisingly thin. Traditional optimization-based explanations suggest that a large initial rate helps the model "jump" out of bad local minima, while a small rate helps it "settle" into the bottom of a bowl without oscillating. However, these explanations often fail to account for the behavior of modern, deep, and non-convex networks. This paper challenges these views, proposing instead that lrDecay acts as a complexity filter: an initially high learning rate prevents the model from memorizing noise, while subsequent decays allow the model to capture increasingly intricate and specific patterns.

The standard learning rate decay strategy and its impact on training error.

The Phenomenon of Learning Stages

In deep learning, the standard "recipe" is to start with a large learning rate and drop it several times during training. As shown in the figure above, these drops correlate with sharp improvements in model performance. Interestingly, the authors demonstrate that in full-batch Gradient Descent (GD), lrDecay provides almost no benefit to final accuracy—a finding that contradicts the traditional "oscillation" theory. This suggests that the "magic" of lrDecay is deeply tied to the stochastic nature of modern training (SGD).

Mechanism Hypothesis: The Complexity Ladder

The core proposal is that the learning rate magnitude is inversely related to the complexity of the patterns a network can learn.

  • High LR as a Noise Filter: A high initial learning rate acts as a regularizer. It prevents the network from "fitting" or memorizing noisy data points, which are essentially high-complexity, low-frequency patterns.
  • Sequential Learning: Neural networks appear to learn in stages. They capture simple, robust features first. Only when the learning rate is reduced can the model "see" and converge on more complex, subtle patterns.
Optimization trajectories on a non-convex landscape.

Experimental Validation

The authors validated their hypothesis using a variety of datasets:

  • PS10 (Pattern Separation 10): A synthetic dataset where "simple" and "complex" patterns are separated into different channels. Results showed that accuracy on complex patterns only spiked after the learning rate was decayed.
  • Noise Robustness: On PS10 with 10% added noise, starting with a high learning rate outperformed a constant small rate, as the latter caused the model to overfit noise early, blocking the learning of actual patterns.
  • Transfer Learning: Patterns learned in the final stages (after the last lrDecay) were found to be the most complex but also the least transferable. This "Transferability Trap" suggests that for pre-training, the optimal checkpoint might be before the final decay.

Practical Implications: AutoDecay

To operationalize these insights, the authors developed "AutoDecay." This algorithm uses an Exponential Decay Moving Average (EDMA) with bias correction to filter noise from the training loss signal. It automatically triggers a learning rate drop only when the loss has truly plateaued. On ImageNet with ResNet-50, AutoDecay achieved a 75.91% top-1 accuracy in 81 epochs, a 10% speedup over the standard 90-epoch manual schedule.

Conclusion

While the "pattern complexity" view is supported by strong empirical evidence, it currently lacks a rigorous mathematical bridge to formal optimization theory. Future work will investigate how other techniques, like Batch Normalization or Weight Decay, interact with this complexity-filtering mechanism. Nonetheless, this perspective provides a more robust explanation for the success of lrDecay in modern deep learning.

Summary updated

Abstract

Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.
BibTeX
@misc{you2019learning,
  author = {Kaichao You and Mingsheng Long and Michael I. Jordan and Jianmin Wang},
  title = {{Learning Stages: Phenomenon, Root Cause, Mechanism Hypothesis, and Implications}},
  year = {2019},
  eprint = {1908.01878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.01878},
}
Figure