Loss Spike in Training Neural Networks

Zhongwang Zhang, Zhi-Qin John Xu

Abstract

This work investigates the mechanism underlying loss spikes observed during neural network training. When training enters a region with a "smaller-loss-as-sharper" (SLAS) structure, it becomes unstable, leading to an exponential increase in loss—the rapid ascent of a loss spike. Stability is regained upon finding a flatter region. We find that deviations in the primary eigen-direction (associated with the maximum Hessian eigenvalue, $\lambda_{max}$) are dominated by low-frequency components. Since neural networks capture low frequencies rapidly (the Frequency Principle), a rapid descent follows the spike. We revisit the link between $\lambda_{max}$ flatness and generalization, finding that because low-frequency information is well-captured by both training and test data, $\lambda_{max}$ is a poor indicator of generalization. Instead, loss spikes facilitate "condensation"—the alignment of input weights—which may be the true mechanism by which spikes improve generalization.

Introduction: The Edge of Stability

Practitioners training deep neural networks often encounter sudden, massive spikes in the loss curve that quickly subside. This phenomenon occurs in a regime known as the Edge of Stability (EoS). While traditional optimization theory suggests that the sharpness of the loss landscape, measured by the maximum eigenvalue of the Hessian ($\lambda_{max}$), should remain below a threshold of $2/\eta$ (where $\eta$ is the learning rate), modern neural networks often hover right at this boundary.

This research addresses three critical questions: What landscape structure causes these spikes? Why does the loss recover so rapidly? And does the resulting "flatness" actually correlate with better generalization?

The Mechanism of Loss Spikes

The authors identify a specific landscape structure called Smaller-Loss-As-Sharper (SLAS) as the primary driver of instability. In these regions, the only path to a lower training loss is through increasingly sharp territory. Eventually, the landscape becomes so sharp that $\lambda_{max}$ exceeds the $2/\eta$ stability limit, causing the optimizer to become unstable and the loss to explode.

Training dynamics showing loss spikes occurring when the maximum eigenvalue $\lambda_{max}$ exceeds the stability threshold $2/\eta$.

As shown in the figure above, loss spikes occur precisely when the sharpness of the landscape crosses the theoretical stability threshold. This "catapult" effect pushes the parameters out of the sharp region into a flatter one where training can resume.

Rapid Descent and the Frequency Principle

A notable feature of these spikes is the rapid descent that follows the initial explosion. The authors explain this using the Frequency Principle, which states that neural networks are biased toward learning low-frequency components of a function much faster than high-frequency ones.

Analysis of the parameter deviations during a spike reveals that the movement in the direction of $\lambda_{max}$ is dominated by low-frequency information. After a spike "catapults" the model into a new region, the network rapidly re-learns these broad, low-frequency features, leading to the observed fast recovery in loss.

Re-evaluating Generalization: Flatness vs. Condensation

The "flatness vs. generalization" debate is central to deep learning theory, with the common intuition being that flatter minima (lower $\lambda_{max}$) generalize better.

Conceptual illustration of flat vs. sharp minima in the loss landscape and their relation to training and test loss.

However, this study challenges that notion. Because the sharpest directions in the Hessian primarily capture low-frequency data—which is typically consistent between training and test sets—the value of $\lambda_{max}$ itself is not a reliable predictor of generalization performance.

Instead, the authors propose that the benefit of loss spikes lies in condensation. During a spike, input weights of different neurons tend to evolve toward the same direction, effectively reducing the effective complexity of the network. This simplified, condensed state is strongly correlated with improved generalization, suggesting that the "cleaning" effect of the spike on the weight distribution is more important than the final flatness of the landscape.

Conclusion

Loss spikes are not mere training artifacts but are fundamental to the optimization dynamics of deep networks. By facilitating condensation, these periods of instability help models reach more robust, generalizable solutions. Future research will explore whether these "controlled instabilities" can be intentionally triggered to improve training outcomes across diverse architectures.

Summary updated

Abstract

In this work, we study the mechanism underlying loss spikes observed during neural network training. When the training enters a region, which has a smaller-loss-as-sharper (SLAS) structure, the training becomes unstable and loss exponentially increases once it is too sharp, i.e., the rapid ascent of the loss spike. The training becomes stable when it finds a flat region. The deviation in the first eigen direction (with maximum eigenvalue of the loss Hessian ($\lambda_{\mathrm{max}}$) is found to be dominated by low-frequency. Since low-frequency is captured very fast (frequency principle), the rapid descent is then observed. Inspired by our analysis of loss spikes, we revisit the link between $\lambda_{\mathrm{max}}$ flatness and generalization. For real datasets, low-frequency is often dominant and well-captured by both the training data and the test data. Then, a solution with good generalization and a solution with bad generalization can both learn low-frequency well, thus, they have little difference in the sharpest direction. Therefore, although $\lambda_{\mathrm{max}}$ can indicate the sharpness of the loss landscape, deviation in its corresponding eigen direction is not responsible for the generalization difference. We also find that loss spikes can facilitate condensation, i.e., input weights evolve towards the same, which may be the underlying mechanism for why the loss spike improves generalization, rather than simply controlling the value of $\lambda_{\mathrm{max}}$.
BibTeX
@misc{zhang2023loss,
  author = {Zhongwang Zhang and Zhi-Qin John Xu},
  title = {{Loss Spike in Training Neural Networks}},
  year = {2023},
  eprint = {2305.12133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.12133},
}
Figure