Abstract
This work investigates the mechanism underlying loss spikes observed during neural network training. When training enters a region with a "smaller-loss-as-sharper" (SLAS) structure, it becomes unstable, leading to an exponential increase in loss—the rapid ascent of a loss spike. Stability is regained upon finding a flatter region. We find that deviations in the primary eigen-direction (associated with the maximum Hessian eigenvalue, $\lambda_{max}$) are dominated by low-frequency components. Since neural networks capture low frequencies rapidly (the Frequency Principle), a rapid descent follows the spike. We revisit the link between $\lambda_{max}$ flatness and generalization, finding that because low-frequency information is well-captured by both training and test data, $\lambda_{max}$ is a poor indicator of generalization. Instead, loss spikes facilitate "condensation"—the alignment of input weights—which may be the true mechanism by which spikes improve generalization.
Introduction: The Edge of Stability
Practitioners training deep neural networks often encounter sudden, massive spikes in the loss curve that quickly subside. This phenomenon occurs in a regime known as the Edge of Stability (EoS). While traditional optimization theory suggests that the sharpness of the loss landscape, measured by the maximum eigenvalue of the Hessian ($\lambda_{max}$), should remain below a threshold of $2/\eta$ (where $\eta$ is the learning rate), modern neural networks often hover right at this boundary.
This research addresses three critical questions: What landscape structure causes these spikes? Why does the loss recover so rapidly? And does the resulting "flatness" actually correlate with better generalization?
The Mechanism of Loss Spikes
The authors identify a specific landscape structure called Smaller-Loss-As-Sharper (SLAS) as the primary driver of instability. In these regions, the only path to a lower training loss is through increasingly sharp territory. Eventually, the landscape becomes so sharp that $\lambda_{max}$ exceeds the $2/\eta$ stability limit, causing the optimizer to become unstable and the loss to explode.
As shown in the figure above, loss spikes occur precisely when the sharpness of the landscape crosses the theoretical stability threshold. This "catapult" effect pushes the parameters out of the sharp region into a flatter one where training can resume.
Rapid Descent and the Frequency Principle
A notable feature of these spikes is the rapid descent that follows the initial explosion. The authors explain this using the Frequency Principle, which states that neural networks are biased toward learning low-frequency components of a function much faster than high-frequency ones.
Analysis of the parameter deviations during a spike reveals that the movement in the direction of $\lambda_{max}$ is dominated by low-frequency information. After a spike "catapults" the model into a new region, the network rapidly re-learns these broad, low-frequency features, leading to the observed fast recovery in loss.
Re-evaluating Generalization: Flatness vs. Condensation
The "flatness vs. generalization" debate is central to deep learning theory, with the common intuition being that flatter minima (lower $\lambda_{max}$) generalize better.
However, this study challenges that notion. Because the sharpest directions in the Hessian primarily capture low-frequency data—which is typically consistent between training and test sets—the value of $\lambda_{max}$ itself is not a reliable predictor of generalization performance.
Instead, the authors propose that the benefit of loss spikes lies in condensation. During a spike, input weights of different neurons tend to evolve toward the same direction, effectively reducing the effective complexity of the network. This simplified, condensed state is strongly correlated with improved generalization, suggesting that the "cleaning" effect of the spike on the weight distribution is more important than the final flatness of the landscape.
Conclusion
Loss spikes are not mere training artifacts but are fundamental to the optimization dynamics of deep networks. By facilitating condensation, these periods of instability help models reach more robust, generalizable solutions. Future research will explore whether these "controlled instabilities" can be intentionally triggered to improve training outcomes across diverse architectures.
Summary updated