Second-order regression models exhibit progressive sharpening to the edge of stability

Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington

Introduction

Recent observations in deep learning have revealed a fascinating phenomenon when training models with large learning rates: the optimization process pushes the curvature of the loss landscape to the absolute limit of what gradient descent can handle. This behavior, known as the "Edge of Stability" (EoS), is preceded by "progressive sharpening," where the landscape's sharpness increases over time.

Standard theoretical frameworks, such as the Neural Tangent Kernel (NTK), treat models as approximately linear and therefore predict constant sharpness, failing to explain these dynamics. This paper demonstrates that EoS and progressive sharpening are not unique quirks of complex neural networks. By analyzing "second-order regression models"—simple quadratic functions—the authors prove that these phenomena are general properties of high-dimensional non-linear systems trained with discrete steps.

The Problem: Beyond Linear Approximations

When training deep neural networks with large learning rates ($\eta$), two distinct phases typically emerge:

  1. Progressive Sharpening: The "sharpness" of the loss landscape, measured by the largest eigenvalue of the Hessian ($\lambda_{max}$), steadily increases.
  2. Edge of Stability (EoS): The sharpness eventually plateaus just below $2/\eta$. This value represents a mathematical "cliff edge"—if the landscape grew any sharper, gradient descent would diverge.

Because NTK theory assumes the predictive function is approximately linear in its parameters, it cannot capture this intrinsically non-linear self-stabilization. To understand why models naturally gravitate toward this cliff edge, the authors propose a minimalist sandbox: a model non-linear enough to exhibit EoS, yet simple enough to solve mathematically.

Methodology: Second-Order Regression and Two-Step Dynamics

The authors introduce second-order regression models. While a standard linear model takes the form $f(\theta) = W\theta$, this proposed model incorporates a quadratic term:

$$f(\theta) = y + G^T\theta + \frac{1}{2}Q(\theta, \theta)$$

This represents the simplest possible non-linear extension of a linear model.

To analyze the chaotic, oscillating behavior that occurs at the Edge of Stability, the authors employ a "two-step dynamics" approach. Rather than tracking the model's state at every single update, they analyze the dynamics across every other update. This perspective smooths out the bouncing behavior inherent to large learning rates, revealing an underlying attractor that steadily pulls the model toward the stability boundary.

Experimental Setup

The researchers validated their theory using a combination of synthetic and real-world datasets:

  • Synthetic Quadratic Models: The authors generated random high-dimensional tensors ($Q$) and matrices ($G$) to test whether EoS behavior is a "generic" property of quadratic systems.
  • CIFAR-10: To ensure these findings translate to real-world deep learning, they trained a standard 2-hidden-layer fully connected network (FCN) on CIFAR-10, comparing its behavior to the quadratic model's predictions.

Key Insights and Results

EoS is a High-Dimensional Non-Linear Phenomenon

The most significant takeaway is that EoS does not require layers, activation functions, or complex architectures. A simple quadratic term paired with a large learning rate is sufficient to trigger progressive sharpening and EoS.

The Power of Two-Cycle Dynamics

By analyzing the dynamics over two iterations, the authors discovered that the system naturally settles into a "two-cycle." While the model appears to be oscillating wildly step-to-step, the two-step view reveals it is actually following a slow, stable path along the edge of the stability boundary.

The Role of Asymmetry and Negative Curvature

For EoS to occur, the model's structure requires a specific asymmetry: at least one large positive eigenvalue and one small negative eigenvalue. This structural imbalance allows the model to sharpen initially before stabilizing.

Predictable Deviations and Genericity

The authors mathematically proved that for a two-parameter model, the system converges to the Edge of Stability across a wide range of initializations. Crucially, the sharpness does not hit exactly $2/\eta$. In their 2D model, it converges to $(2 - \epsilon/2)/\eta$, where $\epsilon$ represents the model's internal non-linearity.

Simulations with hundreds of parameters confirmed that this EoS behavior is "generic"—it occurs almost universally, regardless of the specific random initialization. Furthermore, tests on CIFAR-10 showed that the non-linear feedback predicted by the quadratic model perfectly matched the behavior of the top eigenmode in the real neural network, suggesting that EoS in deep networks is fundamentally driven by low-dimensional quadratic interactions.

Limitations and Future Directions

While second-order regression models successfully capture the onset of EoS, they do not perfectly replicate the long-term oscillations observed in real neural networks. The authors note that cubic or higher-order terms (Taylor expansions beyond the second order) are likely necessary to model the full "wobble" of the training process.

Additionally, this research focuses strictly on optimization dynamics—how the model navigates to a minimum—rather than feature learning. Connecting these quadratic dynamics to how a model develops its internal representations remains an open challenge. Finally, while the exact mathematical deviation from $2/\eta$ is proven for simple models, calculating the precise offset for complex architectures like ResNets or Transformers represents the next frontier in understanding the Edge of Stability.

Summary updated

Abstract

Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive sharpening), followed by a stabilization of the eigenvalue near the maximum value which allows convergence (edge of stability). These phenomena are intrinsically non-linear and do not happen for models in the constant Neural Tangent Kernel (NTK) regime, for which the predictive function is approximately linear in the parameters. As such, we consider the next simplest class of predictive models, namely those that are quadratic in the parameters, which we call second-order regression models. For quadratic objectives in two dimensions, we prove that this second-order regression model exhibits progressive sharpening of the NTK eigenvalue towards a value that differs slightly from the edge of stability, which we explicitly compute. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network, suggesting that progressive sharpening and edge-of-stability behavior aren’t unique features of neural networks, and could be a more general property of discrete learning algorithms in high-dimensional non-linear models.
BibTeX
@inproceedings{agarwala2023second,
  author = {Atish Agarwala and Fabian Pedregosa and Jeffrey Pennington},
  title = {{Second-order regression models exhibit progressive sharpening to the edge of stability}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=9Bz2AoV1l6},
}
Figure