Why Gradient Descent Almost Always Avoids Saddle Points

This direction is merged into another direction: saddle point stagnation, saddle point avoidance. Its last version stays readable and citable here.

Version 1 of 180 papers132 conceptsfrozen 2026-10-04

In the landscape of non-convex optimization, saddle points are theoretically everywhere. For a long time, researchers feared these points would trap gradient-based algorithms, preventing them from reaching local minimizers. However, empirical evidence from deep learning suggested otherwise: simple gradient descent (GD) seemed to navigate these treacherous terrains with ease.

The theoretical explanation for this success lies in a beautiful intersection of dynamical systems and measure theory. It turns out that for almost any random initialization, first-order methods are mathematically "repelled" by saddle points.

The Geometry of Instability

To understand why we don't get stuck, we first need to define the "enemy." A strict saddle point is a stationary point where the Hessian matrix has at least one negative eigenvalue. This means that while the gradient is zero, there is at least one direction where the function value decreases.

As Jason D. Lee, Michael I. Jordan, and their colleagues demonstrated in Gradient Descent Converges to Minimizers, these negative curvature directions are the key to escape. If an algorithm is initialized randomly, it is extremely unlikely to be perfectly aligned with the "stable" directions that lead into the saddle point.

The Stable Manifold Theorem

The core mechanism behind saddle point avoidance is the Stable Manifold Theorem. In the context of an optimization algorithm viewed as a discrete dynamical system, every strict saddle point has a "stable manifold"—a set of initial points that will eventually converge to that saddle.

The breakthrough insight from researchers like Ioannis Panageas and Georgios Piliouras in Gradient Descent Converges to Minimizers: The Case of Non-Isolated Critical Points is that this stable manifold is a lower-dimensional structure.

Imagine a 3D landscape where a saddle point sits like a pass between two mountains. The only way to "get stuck" at the pass is to approach it along a very specific, infinitely thin line. If you start even a fraction of a millimeter to the left or right, the negative curvature will eventually catch you and fling you down toward a valley (a minimizer). In mathematical terms, this thin line has Lebesgue measure zero. Because a random initialization picks a point from the full volume of the space, the probability of landing exactly on that measure-zero line is zero.

The Role of Diffeomorphisms

For this "measure zero" guarantee to hold across multiple iterations, the optimization update rule must behave well. Specifically, the update mapping must be a diffeomorphism—a transformation that is both smooth and invertible.

As explained by Enbin Song and Zhubin Shen in Block Coordinate Descent Only Converge to Minimizers, a diffeomorphism ensures that the algorithm doesn't "collapse" the search space. If the mapping is invertible, it preserves the measure-zero property of the bad initial conditions. This ensures that if the set of "trapping" points started as a thin line, it stays a thin line (or a thin surface) and never expands to fill a volume that could catch a random trajectory.

Universality Across Algorithms

This phenomenon isn't limited to vanilla gradient descent. The same geometric principles apply to a wide array of modern optimization techniques:

Why Randomness is Sufficient

The most practical takeaway from this research direction is that we don't need to inject noise during training to escape saddle points. While stochastic noise (as in SGD) can help, the inherent instability of saddle points combined with a single random initialization is often enough.

As Arnulf Jentzen argues in Gradient Descent Provably Escapes Saddle Points in the Training of Shallow ReLU Networks, even in non-smooth landscapes like those created by ReLU activations, the set of "bad" starting parameters is negligible. The geometry of the loss landscape itself acts as a guide, ensuring that for almost every starting point, the only stable destinations are the ones we actually want: the local minimizers.

Papers in this version

Showing 60 of 80 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_why_gradient_descent_almost_always_avoid_af631a7a,
  title = {Why Gradient Descent Almost Always Avoids Saddle Points},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/why-gradient-descent-almost-always-avoids-saddle-points/rdr_af631a7a0cac4c6290ce9445119d0a01},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_why_gradient_descent_almost_always_avoid_af631a7a_v1,
  title = {Why Gradient Descent Almost Always Avoids Saddle Points},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/why-gradient-descent-almost-always-avoids-saddle-points/rdr_af631a7a0cac4c6290ce9445119d0a01/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}