Optimization Dynamics at the Edge of Stability

25 concepts21 papers3 proposals

In classical optimization theory, gradient descent follows a strict speed limit. If you choose a learning rate ($\eta$) that is greater than $2/\lambda_{\max}$ (where $\lambda_{\max}$ is the maximum eigenvalue of the Hessian matrix, representing the sharpest curvature of the loss landscape), the system is expected to catastrophically diverge. The quadratic approximation predicts that the optimizer will overshoot the minimum, landing higher and higher up the valley walls until the loss explodes.

Yet, modern deep neural networks routinely flout this rule. Practitioners regularly use large learning rates that defy classical stability limits, and instead of blowing up, the models successfully minimize the loss.

This paradox is explained by a phenomenon known as the Edge of Stability (EoS). Rather than operating in a safe, static regime, gradient descent actively drives neural networks to the brink of instability, triggering a self-correcting feedback loop that shapes the geometry of the loss landscape itself.

The Edge of Stability Paradox

As discovered by Cohen et al. in Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, neural networks trained with gradient descent undergo a phase called progressive sharpening. During the early stages of training, the local curvature of the loss landscape steadily increases—the valleys become narrower and sharper.

Eventually, this sharpness hits a hard ceiling: $\lambda_{\max}$ reaches exactly $2/\eta$.

<--- Intuition: Progressive sharpening pushes the model until it hits the stability wall --->

Sharpness (\lambda_max)
   ^
   |          /------------------- Edge of Stability (\lambda_max = 2/\eta)
   |         /
   |        /  <- Progressive Sharpening
   |       /
   +-----------------------------------> Time

Instead of diverging when it crosses this threshold, the model enters a dynamic equilibrium. The training loss begins to oscillate wildly, but over the long term, it continues to decrease. The optimizer hovers right at the edge of stability, bounding the maximum curvature near the theoretical limit dictated by the inverse of the learning rate (Cohen et al.; Chen et al.).

Why It Works: The Self-Correcting Feedback Loop

To understand why the loss does not explode, we must look at what happens when the optimizer overshoots.

Standard optimization steps are often too large relative to the narrowness of high-curvature regions (Jastrzebski et al.). When the learning rate exceeds the stability threshold, a step along the sharpest direction overshoots the valley floor and bounces high up onto the opposite wall. Ahn et al. demonstrated that during these unstable iterations, the gradient direction becomes highly aligned with the top eigenvector of the Hessian.

This alignment creates a powerful negative feedback loop:

  1. Overshooting: The model enters a region where the sharpness exceeds $2/\eta$.
  2. Instability: The optimizer takes an excessively large step, causing the parameters to bounce violently across the narrow valley walls (Jastrzebski et al.).
  3. Deflection: Because neural network loss landscapes are non-quadratic, these massive, oscillatory gradients do not just push the parameters up the walls; they generate a net force that deflects the weights along the valley floor (Wang et al.).
  4. Self-Correction: This lateral movement guides the parameters away from the narrow bottleneck and into wider, flatter regions of the loss landscape where the curvature is lower (Wang et al.; Arora et al.).
   High Curvature (Sharp Valley)          Low Curvature (Flat Valley)
        |        |                             |               |
        |  /\    |                             |               |
        | /  \   |  ===> Instability drives ===>|   \       /   |
        |/    \  |       weights laterally     |    \_____/    |
        |      \/|                             |               |

Arora et al. formalized this mathematically, proving that gradient descent beyond the stability threshold follows a deterministic flow on the manifold of minimizers that implicitly minimizes the largest eigenvalue of the Hessian. The numerical instability itself acts as a regulatory governor, ensuring that the model remains in a regime where the learning rate is just barely sustainable (Wang et al.).

Actively Regulating and Exploiting Curvature

Recognizing that the optimization trajectory is a continuous negotiation with curvature has allowed researchers to demystify classic training heuristics and engineer more efficient optimization algorithms.

Demystifying Heuristics

Many standard deep learning practices—such as learning rate warmup, Batch Normalization, and specialized initializations—were originally discovered through trial and error. Gilmer et al. provided a unifying perspective showing that these strategies succeed precisely because they manage curvature. They ensure the optimization trajectory either stays within stable regimes or quickly navigates early, hyper-sharp regions without exploding. Similarly, Arpit et al. leveraged mean field theory to design robust initializations for Weight Normalization and ResNets, directly preventing the high initial curvature that restricts large learning rates.

Curvature-Aware Optimizers

Instead of passively waiting for the Edge of Stability to push the model into flat regions, new optimization techniques actively exploit this dynamic. Vincent Roulet developed Curvature Dynamics Aware Tuning (CDAT) in Stepping on the Edge: Curvature Aware Learning Rate Tuners. CDAT uses second-order curvature estimates (via Hessian-vector products) to deliberately select a learning rate slightly above the stability threshold. By intentionally provoking mild instability, it forces the landscape to flatten, inducing automatic learning rate warmup and faster convergence. On the other hand, methods like ADLER (Balboni et al.) approximate the Hessian to ensure that step sizes do not overshoot valid local neighborhoods where quadratic approximations break down.

Architectural Disparity

In complex architectures like Transformers, curvature is not distributed evenly. Wang et al. discovered the Sharpness Disparity Principle, revealing that different components of a Transformer (such as the Embedding layers versus LayerNorm) exhibit vastly different levels of sharpness.

When trained with a global learning rate, the entire model's progress is bottlenecked by the sharpest, most sensitive regions. To prevent these sharp regions from diverging, flatter blocks are forced to update at a sub-optimal pace. By decoupling the training dynamics of flat and sharp directions—either through block-specific learning rates (Wang et al.) or via Implicit Regularization Enhancement (Wang et al.)—researchers have achieved up to a 2× speedup in large language model pre-training.

Generalization and Scaling

The implicit bias of gradient descent toward flatter regions has profound implications for how models generalize and scale.

Chen et al. argue that the self-regulating mechanism of the Edge of Stability biases gradient descent toward flatter minima, which are traditionally associated with superior generalization performance. However, this relationship is not entirely straightforward. Kaur et al. challenged the universality of this "flatness hypothesis," demonstrating that while adjusting learning rates systematically manipulates $\lambda_{\max}$, a reduction in this maximum eigenvalue does not universally guarantee better generalization on unseen data.

As the industry scales up to billions of parameters, understanding curvature consistency becomes vital. Noci et al. demonstrated in Super Consistency of Neural Network Landscapes and Learning Rate Transfer that training stability depends on how sharpness behaves across model sizes. If the loss landscape's sharpness remains consistent as a model scales, hyperparameter tuning becomes predictable, allowing optimal learning rates to be seamlessly transferred from small prototype models to massive production networks.

Finally, rather than relying solely on the optimizer to find flat regions, some approaches alter the landscape itself. Ruiz-Garcia et al. introduced "dynamical loss functions" that cyclically shift focus between different classes during training. This "peristaltic" movement of the loss landscape mechanically forces the parameters out of narrow, poorly-generalizing valleys and into deeper, wider minima, demonstrating that manipulating the playing field can be just as effective as changing how the optimizer runs on it.

Key Foundational Work

On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2019) — contributed 2 concept elements to this direction.
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021) — contributed 2 concept elements to this direction.
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 22 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_loss_landscape_curvature_81eac0b3,
  title = {loss landscape curvature},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/loss-landscape-curvature/rdr_81eac0b38c1d4a48bbeaa1ca2ada10bd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_loss_landscape_curvature_81eac0b3_v2,
  title = {loss landscape curvature},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/loss-landscape-curvature/rdr_81eac0b38c1d4a48bbeaa1ca2ada10bd/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake