Gradient Descent Dynamics at the Edge of Stability

129 concepts85 papers1 proposal

In classical optimization theory, there is a strict speed limit for gradient descent. The textbook descent lemma states that for a function with Lipschitz continuous gradients, a constant learning rate $\eta$ guarantees a monotonic decrease in loss if $\eta < 2/L$, where $L$ represents the maximum curvature or "sharpness" of the loss landscape. Geometrically, this sharpness is measured by the maximum eigenvalue of the loss Hessian, $\lambda_{\max}(H)$. If the learning rate exceeds this threshold ($\eta > 2/\lambda_{\max}(H)$), standard mathematical models predict catastrophic divergence—the optimization steps should overshoot the local valley and explode to infinity.

Yet, modern deep neural networks routinely defy this speed limit. Practitioners consistently use aggressive learning rates that violate classical stability bounds, and the networks still converge to highly effective solutions.

The Phenomenon of Progressive Sharpening

To understand this discrepancy, Jeremy Cohen et al. tracked the exact evolution of the Hessian's largest eigenvalue during training. They discovered that neural network optimization does not remain in the safe, low-curvature regimes predicted by classical theory. Instead, it undergoes a systematic process called progressive sharpening: during the early phases of training, the loss landscape becomes steadily sharper, causing $\lambda_{\max}(H)$ to climb.

Eventually, this sharpness hits a hard ceiling at exactly the theoretical stability limit:

$$\lambda_{\max}(H) \approx \frac{2}{\eta}$$

Instead of exploding into divergence upon crossing this threshold, the optimizer enters a regime known as the Edge of Stability (EoS). In this state, the sharpness hovers dynamically around $2/\eta$, and the training loss begins to oscillate wildly over short intervals. Crucially, despite these local, non-monotonic spikes, the loss continues to decrease over longer timescales.

The Mechanics of Self-Stabilization

Why does the optimization process survive this apparent instability? The answer lies in the discrete-time geometry of the optimization path and the non-linearities of deep architectures.

When the learning rate exceeds the stability threshold, a single gradient step overshoots the bottom of the local valley, landing high up on the opposite wall. As analyzed by Lei Chen et al., rather than flying off to infinity, the optimization process often settles into stable period-2 orbits—perpetually jumping back and forth across the valley.

This bounded oscillation is made possible by the relationship between the step size and the directional smoothness of the loss landscape. Kwangjun Ahn et al. demonstrate that when the optimizer overshoots into a sharper region, the higher-order derivatives (the third- and fourth-order shapes of the landscape) act as an automatic brake. The non-linear structure of the network alters the local curvature, pushing the parameters back toward the center of the orbit rather than allowing them to diverge.

This geometric behavior fundamentally alters the trajectory's direction. Shuo Chen et al. show that below the critical learning rate threshold, the optimizer follows a smooth, monotonic path aligned with the direction of minimum curvature. Above it, the path enters an oscillatory regime where it alternates precisely along the direction of maximum curvature. This highly structured oscillation allows the model to self-stabilize, maintaining a macro-level descent despite micro-level chaos.

This discrete-time behavior highlights a fundamental limitation of classical continuous-time analysis (such as gradient flow differential equations). As noted by Kangqiao Liu et al., continuous-time frameworks imply infinite stability and completely miss the massive, distorted fluctuations introduced by a finite learning rate. To capture these phase transitions precisely in high dimensions, Yue Lu employs deterministic scaling limits to map out the exact boundaries where large steps cause a system to either transition into a new optimization phase or collapse entirely.

The Implicit Bias of Instability

Operating at the edge of stability is not merely a theoretical curiosity; it acts as a powerful selection mechanism that guides neural networks toward better-generalizing solutions. In over-parameterized models, there are infinitely many configurations of weights that can achieve zero training error, but they differ drastically in their local geometry.

The large learning rate acts as a dynamical filter. As early work by Kamil Nar et al. and Rotem Mulayoff et al. pointed out, if a local minimum is too sharp ($\lambda_{\max}(H) > 2/\eta$), it becomes dynamically unstable. The optimizer literally cannot settle there; it is violently ejected. Consequently, the instability forces the parameters to wander along the manifold of minimizers until they find a broad, low-curvature valley that can accommodate the large step size.

This process biases gradient descent toward flatter minima. Alex Damian et al. and Sanjeev Arora et al. formalize this implicit bias, proving that the deterministic flow under EoS dynamics actively minimizes sharpness over time. Furthermore, this unstable exploration enables the network to escape shallow local traps and learn complex features—such as those required to model threshold neurons—that remain entirely inaccessible under a conservative, small-step gradient flow (Kwangjun Ahn et al.).

Mathematical Foundations in Simplified Models

To rigorously prove these dynamics without the overwhelming complexity of full-scale networks, researchers analyze simplified mathematical abstractions:

  • Minimalist Networks: Xingyu Zhu et al. and Liming Liu et al. utilize deep scalar and two-layer networks to demonstrate that progressive sharpening and self-stabilization are direct mathematical consequences of network depth and non-linear parameter trajectories.
  • Deep Matrix Factorization: Avrajit Ghosh et al. show that operating past the edge of stability breaks classical symmetry-induced conservation laws, forcing the network's oscillations into a low-dimensional subspace of the most prominent features while driving the overall model toward flatter regions.
  • Linear Models: When extended to over-parameterized least squares, Lachlan Ewen MacDonald et al. categorize the optimization into three distinct regimes: subcritical linear convergence to a flat minimum, critical power-law convergence to the flattest possible minimum, and supercritical linear convergence to a stable oscillating orbit.

Even in classification tasks like logistic regression on separable data, entering the EoS regime via large stepsizes can radically accelerate training. Ruiqi Zhang et al. prove that intentionally violating the descent lemma allows the algorithm to achieve an arbitrarily fast convergence rate by exploiting non-monotonic optimization paths. This demonstrates that minimizing a smooth loss metric step-by-step is not a prerequisite for rapid global convergence; rather, operating at the brink of divergence is a core characteristic of effective deep learning optimization.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_edge_of_stability_070fd47b,
  title = {edge of stability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/edge-of-stability/rdr_070fd47bbf404a56a638eb012b1e04b7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_edge_of_stability_070fd47b_v1,
  title = {edge of stability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/edge-of-stability/rdr_070fd47bbf404a56a638eb012b1e04b7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake