Curvature Dependent Noise and Flatness Selection

This direction is split into other directions: anisotropic gradient noise. Its last version stays readable and citable here.

Version 1 of 189 papers148 conceptsfrozen 2026-10-04

In the landscape of deep learning, Stochastic Gradient Descent (SGD) is often described as a "noisy" version of gradient descent. For years, this noise was viewed as a necessary evil—a byproduct of using mini-batches that we hoped wouldn't derail convergence. However, a growing body of research suggests that the specific structure of this noise is actually the "secret sauce" behind why neural networks generalize so well.

The core insight is that SGD noise is not isotropic (uniform in all directions). Instead, it is geometry-aware. The noise covariance is naturally aligned with the local curvature of the loss landscape, creating a dynamical force that systematically pushes the model away from "sharp" minima and traps it in "flat" ones.

The Geometry of Stochastic Noise

In a deterministic world, an optimizer simply follows the steepest path down. But in SGD, the update at each step is a random variable. Zhanxing Zhu and others have shown that this noise is anisotropic, meaning its intensity varies depending on the direction in parameter space.

Crucially, this noise is not random; it is shaped by the gradient matrix. Zeke Xie, Issei Sato, and Masashi Sugiyama demonstrate in A Diffusion Theory For Deep Learning Dynamics that the noise covariance is approximately proportional to the Hessian (the matrix of second derivatives).

This alignment creates a powerful feedback loop:

  1. In Sharp Regions: Where curvature is high, the noise magnitude is also high. This "shakes" the optimizer violently, making it nearly impossible for the model to stay in a narrow, steep pit.
  2. In Flat Regions: Where curvature is low, the noise is suppressed. The optimizer becomes "quiet," allowing the parameters to settle and stabilize.

This mechanism explains why SGD exponentially favors flat minima. As Lei Wu and Mingze Wang prove in their stability analysis, any minimum that SGD can reliably stay in must have a Hessian norm bounded by the ratio of the learning rate to the batch size.

Entropic Confinement and Barriers

If flat regions are so much better, why don't models simply walk between them? Luca Di Carlo, Chase Goddard, and David J. Schwab identify a phenomenon called entropic confinement in Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks.

Even when two minima are connected by a "low-loss path," the path itself might be narrower than the minima it connects. Because the noise is higher in the narrower (higher curvature) parts of the path, it acts as a statistical "force" that pushes the model back into the wider basins. These are not energetic barriers (the loss isn't higher), but entropic barriers—the optimizer simply "prefers" the volume of the flatter region.

Escaping Saddles via Correlated Curvature

This geometry-aware noise doesn't just help find flat minima; it also helps the model escape "saddle points" (areas where the landscape is flat in some directions but slopes down in others).

Traditional theory suggested we might need to add artificial noise to escape these traps. However, Hadi Daneshmand introduced the concept of Correlated Negative Curvature (CNC) in Escaping Saddles with Stochastic Gradients. The inherent noise in SGD is already "correlated" with the directions of steepest descent. Because the noise is strongest where the landscape is most curved, it naturally provides a "kick" in the exact direction needed to exit a saddle point. Long Yang and colleagues later extended this to show how policy gradients in reinforcement learning use this same property to find second-order stationary points.

The Illusion of the Tiny Subspace

A common observation in deep learning is that gradients seem to "live" in a tiny subspace of high curvature. While this was once thought to be where learning happens, Chulhee Yun, Kwangjun Ahn, and Minhak Song argue in Does SGD really happen in tiny subspaces? that this might be a mechanical artifact.

When noise pushes a model up the steep "walls" of a valley, the resulting gradient points sharply back toward the valley floor. This creates a massive gradient component in the direction of high curvature. However, this component is just a corrective force—it's the model "bouncing" off the walls. The actual learning (progress along the valley floor) happens in the "bulk" subspace where curvature is low, even though the gradient signal there is much smaller.

Engineering the Noise

Understanding that noise structure matters more than noise magnitude has led to new optimization strategies. For example:

By viewing SGD not as a flawed version of gradient descent, but as a geometry-aware dynamical system, we can finally explain why "noisy" training leads to such robust and generalizable models.

Papers in this version

Showing 60 of 89 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_curvature_dependent_noise_and_flatness_s_b10df5e1,
  title = {Curvature Dependent Noise and Flatness Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curvature-dependent-noise-and-flatness-selection/rdr_b10df5e15fc443d3a056dcf6856e91df},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_curvature_dependent_noise_and_flatness_s_b10df5e1_v1,
  title = {Curvature Dependent Noise and Flatness Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curvature-dependent-noise-and-flatness-selection/rdr_b10df5e15fc443d3a056dcf6856e91df/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}