SGD at the Edge of Stability: The Stochastic Sharpness Gap

Fangshuo Liao, Afroditi Kolomvaki, Anastasios KyrillidisarXiv 2026 · arXiv

TL;DR

While full-batch gradient descent trains neural networks at the "Edge of Stability" (EoS)—where the maximum sharpness of the loss landscape hovers precisely at $2/\eta$—mini-batch SGD mysteriously stabilizes at a lower sharpness. This paper provides the first rigorous theoretical explanation for this phenomenon, demonstrating that mini-batch gradient noise acts as an implicit sharpness-suppressing force. The authors derive a closed-form formula for this "stochastic sharpness gap," proving that smaller batch sizes inherently drive the optimizer toward flatter minima.

What Problem Does This Paper Address?

When training deep neural networks with a learning rate $\eta$, classical optimization theory suggests that the training will diverge if the sharpness (the maximum eigenvalue of the Hessian matrix) exceeds $2/\eta$. However, modern networks routinely defy this rule. In full-batch Gradient Descent (GD), networks undergo progressive sharpening until they hit this $2/\eta$ threshold and hover there non-monotonically—a regime known as the Edge of Stability (EoS).

Yet, when practitioners switch to mini-batch Stochastic Gradient Descent (SGD), an unexplained discrepancy emerges: the sharpness plateaus below $2/\eta$, and smaller batch sizes lead to even flatter solutions. Why does stochastic noise suppress full-batch sharpness, and can we mathematically predict this gap? This paper resolves this open problem by analyzing the interaction between gradient noise and the third-order (cubic) geometry of the loss landscape.

Methods

The authors extend the deterministic self-stabilization framework of prior literature to the stochastic setting through several key steps:

  1. Stochastic Predicted Dynamics: They model how gradient noise injects variance into the oscillatory behavior of SGD along the top eigenvector of the Hessian.
  2. Stochastic Coupling Theorem: They prove a mean-square coupling theorem showing that the actual, chaotic SGD trajectory can be tightly approximated by a simplified, constrained projected gradient descent (PGD) trajectory combined with their stochastic predicted dynamics.
  3. Closed-Form Derivation: By analyzing the stationarity conditions of these dynamics under a "frozen-coefficient" regime (where landscape properties are assumed locally constant), they derive an explicit formula for the equilibrium sharpness gap ($\Delta S$): $$\Delta S = \frac{\eta \beta \sigma_{\boldsymbol{u}}^{2}}{4\alpha}$$ Where $\eta$ is the learning rate, $\beta$ is the self-stabilization strength, $\sigma_{\boldsymbol{u}}^{2}$ is the variance of the gradient noise projected onto the top eigenvector, and $\alpha$ is the progressive sharpening rate.

Datasets

The theoretical predictions were empirically validated using:

  • CIFAR-10: Specifically, a standardized subset of 5,000 training images to observe clean, long-term EoS dynamics across multiple architectures.

No new datasets or benchmarks were introduced.

Key Insights and Ideas

  • Noise as a Stabilizing Force: The core intuition is that gradient noise pushes the network's parameters back and forth along the sharpest direction. Because the loss landscape has a cubic structure, these noise-driven oscillations amplify the restoring force that pushes the sharpness down, shifting the equilibrium below $2/\eta$.
  • Predictable Power-Law Scaling: Since mini-batch gradient noise variance ($\sigma_{\boldsymbol{u}}^{2}$) scales inversely with batch size ($1/b$), the derived formula explicitly predicts that the sharpness gap follows a power-law relationship: $\Delta S \propto 1/b$.
  • Resolving the Paradox of Batch Sharpness: The paper clarifies a subtle distinction: while full-batch sharpness is suppressed in SGD, the batch sharpness (the curvature experienced by the mini-batch itself) actually does saturate right at $2/\eta$. This explains how SGD can simultaneously remain stable and operate at the edge of stability.

Key Results

  • Architecture-Wide Validation: The authors verified their sharpness gap predictions across MLPs, CNNs, and ResNets. As shown in Figure 1, smaller batch sizes consistently lead to lower equilibrium sharpness plateaus.
Figure 1: Sharpness λmax(∇2L) during training for different batch sizes. (A) FC-Tanh + MSE: smaller batch sizes produce markedly lower equilibrium sharpness. The dashed black curve shows full-batch GD. (B) CNN + MSE: the same pattern holds with a convolutional architecture, with an even larger gap (Seq ≈131 at b = 50 vs. SGD ≈202). Shaded regions indicate ±1 standard deviation across 5 seeds.
  • Quantitative Agreement on Scaling: On a log-log scale, the theory predicts a slope of $-1$ for the sharpness gap versus batch size. Empirically, the authors observed a fitted slope of $-1.27$ for an FC-Tanh network, demonstrating strong quantitative alignment with their theoretical framework.
  • Amplified Effects in CNNs: The sharpness suppression was particularly dramatic in convolutional networks. For a CNN trained with MSE loss, full-batch GD stabilized at a sharpness of $\approx 202$, whereas SGD with a batch size of 50 stabilized at $\approx 131$—resulting in a massive sharpness gap of $\Delta S \approx 71$.

Limitations and Future Work

  • Measurement Challenges for $\alpha$: In practice, measuring the progressive sharpening coefficient $\alpha$ at the exact theoretical reference trajectory is difficult because the actual SGD path constantly overshoots the stability threshold, leading to noisy or negative empirical measurements of $\alpha$.
  • Loss Function Sensitivity: The stochastic self-stabilization mechanism requires a sustained EoS regime to operate. When the authors swapped the MSE loss for Cross-Entropy (CE) loss on the small CIFAR-10 subset, the network rapidly entered an interpolation regime where the loss approached zero and the Hessian flattened out completely, causing the batch-size dependence to disappear. Developing practical methods to measure these landscape quantities dynamically remains an open avenue for future research.

Summary updated

Abstract

When training neural networks with full-batch gradient descent (GD) and step size $η$, the largest eigenvalue of the Hessian -- the sharpness $S(\boldsymbolθ)$ -- rises to $2/η$ and hovers there, a phenomenon termed the Edge of Stability (EoS). \citet{damian2023selfstab} showed that this behavior is explained by a self-stabilization mechanism driven by third-order structure of the loss, and that GD implicitly follows projected gradient descent (PGD) on the constraint $ S(\boldsymbolθ)\leq 2/η$. For mini-batch stochastic gradient descent (SGD), the sharpness stabilizes below $2/η$, with the gap widening as the batch size decreases; yet no theoretical explanation exists for this suppression. We introduce stochastic self-stabilization, extending the self-stabilization framework to SGD. Our key insight is that gradient noise injects variance into the oscillatory dynamics along the top Hessian eigenvector, strengthening the cubic sharpness-reducing force and shifting the equilibrium below $2/η$. Following the approach of \citet{damian2023selfstab}, we define stochastic predicted dynamics relative to a moving projected gradient descent trajectory and prove a stochastic coupling theorem that bounds the deviation of SGD from these predictions. We derive a closed-form equilibrium sharpness gap: $ΔS = ηβσ_{\boldsymbol{u}}^{2}/(4α)$, where $α$ is the progressive sharpening rate, $β$ is the self-stabilization strength, and $σ_{ \boldsymbol{u}}^{2}$ is the gradient noise variance projected onto the top eigenvector. This formula predicts that smaller batch sizes yield flatter solutions and recovers GD when the batch equals the full dataset.
BibTeX
@misc{liao2026sgd,
  author = {Fangshuo Liao and Afroditi Kolomvaki and Anastasios Kyrillidis},
  title = {{SGD at the Edge of Stability: The Stochastic Sharpness Gap}},
  year = {2026},
  eprint = {2604.21016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.21016},
}

Loading related content...

Figure