In classical optimization theory, there is a strict "speed limit" for gradient descent. If you are optimizing a function using a learning rate $\eta$, the step size must satisfy $\eta < 2/\lambda_{\max}$, where $\lambda_{\max}$ is the local sharpness (the maximum eigenvalue of the Hessian matrix). If the learning rate exceeds this threshold, the optimization algorithm is mathematically expected to diverge, causing the loss to explode.
Yet, modern deep neural networks routinely break this rule. As Jeremy Cohen et al. demonstrated in Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, neural networks trained with gradient descent undergo a phenomenon called progressive sharpening: the local curvature of the loss landscape steadily increases during the early-to-mid phases of training. This sharpening continues until it hits the critical stability threshold where $\lambda_{\max} \approx 2/\eta$.
Instead of exploding, the network enters a regime known as the Edge of Stability (EoS). In this state, the model hovers right at the brink of divergence, with the loss oscillating wildly over short intervals but ultimately continuing to decrease over the long term. Understanding why networks naturally seek out these sharp regions—and how they survive them—has become a central focus of deep learning optimization research.
The Drive Toward High Curvature
Why do neural networks actively migrate away from flat, safe regions and into highly curved, theoretically unstable valleys?
One compelling explanation lies in the geometric structure of deep learning loss landscapes. Zhongwang Zhang in Loss Spike in Training Neural Networks identifies a recurring landscape property termed "Lower-Loss-As-Sharper" (LLAS). In many architectures, the path toward lower training error is intrinsically correlated with an increase in local sharpness. The optimizer does not choose sharpness for its own sake; rather, the geometric route to minimizing loss forces the parameters into increasingly steep terrain.
This behavior is heavily influenced by optimization hyperparameters. Early work by Stanislaw Jastrzebski et al. in On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length showed that the maximum curvature attained during early training is tightly constrained by the ratio of the learning rate to the batch size. Furthermore, Lachlan Ewen MacDonald et al. suggest in On progressive sharpening, flat minima and generalisation that this sharpening is fundamentally tied to the smoothness of the input-output Jacobian, indicating that the network's functional complexity and its landscape curvature evolve in tandem.
Surviving the Edge: Mechanisms of Self-Stabilization
When the local sharpness $\lambda_{\max}$ surpasses $2/\eta$, the gradient steps become too large for the narrow valley the model is traversing. Instead of tracking smoothly down the center of the valley, the optimizer begins to overshoot.
To understand why this does not lead to catastrophic failure, we must look at the fine-grained trajectory of the parameters. Kwangjun Ahn et al. in Understanding the unstable convergence of gradient descent explain that as the model enters the unstable regime, the optimization steps align themselves almost entirely with the directions of highest curvature. This alignment creates a dynamic equilibrium that pins the model to the boundary of stability.
Rather than escaping the valley or blowing up to infinity, the trajectory forms what Lei Chen et al. describe in On Gradient Descent Convergence beyond the Edge of Stability as "period-2 orbits." The parameters bounce back and forth across the steep walls of the valley. Each oversized step climbs up one wall, generating a massive restoring gradient that flings the model back across to the opposite wall on the next step.
Alex Damian et al. formalize this behavior in Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. They show that this oscillatory motion exerts an implicit regularization effect. The local instability prevents the model from settling into the sharpest sub-regions, effectively forcing the trajectory to navigate along the ridge of the stability boundary until it finds a path toward flatter, more stable minima.
Deconstructing Dynamics via Minimalist Models
While these dynamics are observed in massive models like ResNets and Transformers, their mathematical complexity makes global proofs difficult. To build rigorous intuition, researchers have recently turned to minimalist models that isolate these exact behaviors.
- Routes to Chaos: Dayal Singh Kalra et al. in Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos utilize a two-layer linear model to show that early-stage flattening and late-stage EoS are universal mathematical properties. By increasing the learning rate, they map out a literal "route to chaos," where simple stable trajectories fracture into complex, non-linear oscillations.
- Steering to Distant Minima: Liming Liu et al. in A Minimalist Example of Edge-of-Stability and Progressive Sharpening analyze a two-layer network with a two-dimensional input. They prove that large learning rates naturally reject nearby sharp minima, driving the model through unstable oscillations until it lands in more distant, flatter regions.
- Architectural Determinants: Using a single-neuron-per-layer setup, Geonhui Yoo et al. in Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More demonstrate how dataset difficulty and network depth explicitly dictate the rate of progressive sharpening and the final stabilization ceiling.
Practical Implications for Training
Understanding the interaction between progressive sharpening and the edge of stability allows practitioners to manipulate training dynamics more effectively.
For example, Polyak's momentum is widely known to accelerate training, but its geometric effect on the landscape was long unmapped. Junghyun Lee et al. in Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study discovered that momentum acts as a powerful lens during EoS. When the model hits the instability threshold, momentum triggers a massive "catapult" event, violently ejecting the parameters from high-curvature zones into significantly flatter regions of the loss landscape.
Alternatively, rather than waiting for the model to self-stabilize naturally, algorithms can actively exploit these dynamics. Mingze Wang et al. in Improving Generalization and Convergence by Enhancing Implicit Regularization introduce a framework that decouples the optimization steps along flat and sharp directions. By deliberately managing landscape sharpness and enhancing the optimizer's implicit regularization, they demonstrate that models can achieve faster convergence and better generalization without suffering from the disruptive, wild oscillations typical of the unmanaged Edge of Stability.
Key Foundational Work
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2020) — contributed 1 concept elements to this direction.
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2019) — contributed 1 concept elements to this direction.