How Does Learning Rate Decay Help Modern Neural Networks?

Kaichao You, Mingsheng Long, Jianmin Wang, Michael I. Jordan

Abstract

Learning rate decay (lrDecay) is a de facto technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

Introduction

In the landscape of deep learning, learning rate decay (lrDecay) is a ubiquitous strategy. The standard approach—starting with a high learning rate and reducing it by a factor of 10 at fixed intervals—is known to provide significant performance gains. While textbook explanations often attribute this to optimization stability, a recent study from researchers at Tsinghua University and UC Berkeley suggests that this optimization-centric view misses a more fundamental mechanism related to data complexity.

The standard learning rate decay strategy (left) and the resulting 'boost' in error reduction observed in ResNet architectures (right).

Challenging Traditional Beliefs

Traditional wisdom holds that a high initial learning rate helps the model escape local minima or speed up initial training, while decaying the rate prevents oscillation around the global minimum. However, empirical evidence shows these explanations are incomplete for deep, non-convex models. For instance, when training WideResNets with pure Gradient Descent (GD) instead of Stochastic Gradient Descent (SGD), the characteristic accuracy "boost" after decay is nearly absent. This indicates that the benefit of lrDecay is deeply tied to the stochastic nature of training and its interaction with the data, rather than just convergence stability.

The Pattern Complexity Hypothesis

The researchers propose a novel explanation: lrDecay acts as a curriculum for "pattern complexity." They formalize this idea by suggesting that neural networks learn simple, high-gain patterns first, and only move to more complex features when the learning rate is reduced.

  • Noise Filtering: A large initial learning rate acts as a powerful regularizer, preventing the model from memorizing low-gain noise early in the training process. Starting with a small learning rate can lead to premature overfitting of noise, harming final generalization.
  • Complexity Curriculum: Neural networks naturally master "easy" patterns first. When the loss curve flattens, it signifies that the model has learned all it can at the current learning rate. Decaying the rate allows the network to resolve and learn more intricate, complex patterns.
Optimization trajectories on a loss landscape, illustrating how different learning rates affect the path taken by the model.

Empirical Evidence and Results

The study validates this hypothesis using several benchmarks:

  • PS10 Dataset: A custom dataset designed with tractable pattern complexity. Experiments showed that simple patterns are learned almost immediately, whereas complex patterns only show an accuracy surge following a learning rate decay.
  • Transferability Analysis: The authors found that features learned in the final stages of training (Stage 3) are significantly less transferable to new datasets (e.g., CUB-200, MITIndoors) than those learned in Stage 2. This supports the idea that later stages focus on highly specific, complex patterns.
  • AutoDecay Efficiency: Leveraging these insights, the researchers developed AutoDecay, which triggers decays based on loss plateaus. This method reached 75.91% top-1 accuracy on ImageNet in 81 epochs, outperforming the standard 90-epoch schedule.

Conclusion

By shifting the focus from optimization to pattern complexity, this research provides a more robust explanation for the effectiveness of learning rate decay. This perspective not only clarifies why traditional explanations fail for modern architectures but also paves the way for more efficient, data-driven training strategies like AutoDecay.

Summary updated

Abstract

Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.
BibTeX
@misc{you2019how,
  author = {Kaichao You and Mingsheng Long and Jianmin Wang and Michael I. Jordan},
  title = {{How Does Learning Rate Decay Help Modern Neural Networks?}},
  year = {2019},
  eprint = {1908.01878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.01878},
}
Figure