Abstract
Standard optimization theory often advocates for small learning rates to ensure stable convergence. However, empirical success in deep learning frequently relies on large initial learning rates. This paper investigates the mechanism behind this phenomenon, proposing that large step sizes allow Gradient Descent (GD) to physically "skip" over sharp local minima. Through theoretical analysis and a novel "SGD with Repeats" experimental framework, the authors demonstrate that this geometric skipping effect is a fundamental property of discrete-time optimization that cannot be explained by stochastic noise alone.
Introduction
In deep learning practice, the choice of learning rate (LR) is critical. While small LRs offer stability, large LRs are often necessary to achieve state-of-the-art performance. The common explanation is that large LRs increase the noise in Stochastic Gradient Descent (SGD), helping the optimizer escape poor local minima. This research challenges that view by asking if the magnitude of the step size itself—independent of gradient noise—provides a unique advantage. The authors explore whether the discrete nature of GD allows it to traverse loss landscapes in ways that continuous-time models, like Gradient Flow, cannot capture.
Methodology
The study employs a multi-faceted approach to isolate the effects of step size:
- Theoretical Framework: The authors define a class of non-convex functions ($C_l$) to prove that GD with a sufficiently large LR can escape local minima that trap smaller-step optimizers.
- SGD with Repeats: To decouple step size from noise, researchers compared standard SGD with a version using a smaller LR but repeating the same mini-batch multiple times. This ensures the total gradient bias remains constant while the physical displacement in weight space varies.
- Trajectory Visualization: High-dimensional training trajectories were projected into 2D using Principal Component Analysis (PCA) to observe how LR choices dictate the regions of the loss landscape explored by the model.
- Synthetic Landscapes: Custom 1D and 2D functions were designed to model sharp local minima and flatter global regions, providing a controlled environment to observe optimization dynamics.
Key Insights: The Skipping Phenomenon
The central finding is the "skipping" phenomenon. When an optimizer encounters a sharp (narrow) local minimum, a large step size can land the next iterate on the opposite side of the "hole," effectively ignoring the obstacle. This suggests that the discreteness of GD is a beneficial feature rather than a numerical limitation.
As illustrated in the figure above:
- Scenario (a): A large LR allows the optimizer to jump over a sharp minimum that traps a small LR optimizer.
- Scenario (b): In certain geometries, both step sizes may escape.
- Scenario (c): If the step is not large enough relative to the width of the minimum, both may become trapped.
The "SGD with Repeats" experiment confirmed that even when noise levels are matched, the version with the larger step size achieves significantly higher accuracy. This implies an "implicit bias" inherent to large steps that noise cannot replicate.
Key Results
The paper provides both formal and empirical evidence for its claims:
- Formal Proofs: Theorems establish that the probability of avoiding sharp local minima increases monotonically with the learning rate.
- Empirical Performance Gap: On CIFAR-10, standard large-LR SGD (0.01) significantly outperformed the noise-matched small-LR version (0.001 with 10 repeats). The repeats helped slightly but could not close the performance gap.
- Irreversibility: The authors observed that once a large LR pushes the optimizer into a favorable region, the trajectory remains there even if the LR is subsequently reduced, indicating a fundamental shift in the optimization path.
Conclusion and Future Work
This research highlights that large learning rates do more than just amplify noise; they fundamentally alter the optimizer's ability to navigate complex, non-convex landscapes by skipping over sharp minima. While the study relies on specific landscape assumptions like One-Point Strong Convexity (OPSC), it opens new avenues for designing optimizers based on their "escaping" properties. Future work should further investigate the direct link between this skipping behavior and the generalization capabilities of the resulting models.
Summary updated