Reconciling Optimization Theory with Non-Convex Loss Landscapes

This direction is split into other directions: non-convex convergence theory. Its last version stays readable and citable here.

Version 1 of 11,084 papers1,158 conceptsfrozen 2026-10-04

The mathematical foundation of deep learning is built on a paradox. Most of the theoretical guarantees we rely on—convergence rates, stability bounds, and optimal learning rates—are derived from the world of convex optimization. In a convex world, every local minimum is a global minimum, and the "landscape" of the loss function is a smooth, predictable bowl.

However, modern neural networks operate in a "messy" reality. As Hoang Tran and Qinzi Zhang point out in Reevaluating Theoretical Analysis Methods for Optimization in Deep Learning, the loss landscapes of deep networks are high-dimensional, non-convex, and often jagged. This creates a significant gap: our optimizers work brilliantly in practice, but the theories explaining why they work often rely on assumptions that are demonstrably false.

The Failure of Classical Assumptions

In classical theory, two pillars support most proofs: convexity (the function curves upward everywhere) and smoothness (the gradient doesn't change too abruptly). When these hold, we can guarantee that an algorithm like Gradient Descent will find the bottom of the bowl.

In deep learning, these pillars crumble:

  • Non-Convexity: Neural networks are riddled with saddle points and local minima. Krishna Reddy Kesari proved that finding the global minimum of a general non-convex function is an exponential-time problem—a "needle-in-a-haystack" scenario that should, theoretically, make deep learning impossible (First Order Methods take Exponential Time to Converge to Global Minimizers of Non-Convex Functions).
  • Jaggedness: While theory often assumes smoothness, real-world training involves "sharp" regions. Hoang Tran found that while smoothness-based theories largely fail to describe practice, convexity-based math surprisingly holds up better than expected, acting as a "good enough" approximation for why models converge.

Bridging the Gap: New Theoretical Frameworks

To move past these restrictive assumptions, researchers are developing new ways to characterize the non-convex journey of an optimizer.

From Points to Manifolds

Instead of looking for a single "optimal point," Benjamin Fehrman, Arnulf Jentzen, and Benjamin Gess suggest we should view convergence as a movement toward manifolds of minima. In Convergence rates for the stochastic gradient descent method for non-convex objective functions, they provide rigorous proofs for SGD that don't require global convexity, better reflecting how models actually settle into stable regions.

The Power of Expressivity

In the realm of meta-learning, the non-convexity problem is even more acute because we are optimizing the optimization process itself. Lingxiao Wang and Zhaoran Wang showed in On the Global Optimality of Model-Agnostic Meta-Learning that if a neural network is "sufficiently expressive," the stationary points found by MAML are actually approximately globally optimal. This suggests that the architecture's capacity can sometimes "overcome" the theoretical difficulties of the non-convex landscape.

Discrete Jumps vs. Continuous Flow

Traditional theory often models gradient descent as a smooth, continuous "flow." However, Amirkeivan Mohtashami argues in Special Properties of Gradient Descent with Large Learning Rates that the discreteness of our steps is a feature, not a bug. Large learning rates allow the optimizer to physically "jump" over sharp local minima that a continuous flow would get stuck in. This "escaping mechanism" is a purely non-convex phenomenon that convex theory completely misses.

Implications for Robustness and Scaling

This theoretical disconnect isn't just an academic concern; it affects how we build robust and efficient systems.

  • Adversarial Training: Standard proofs for stabilizing GANs or adversarial training often assume "convex-concave" surfaces. Abhay Kumar Yadav and Sohil Shah found that these guarantees fail in deep learning, necessitating empirical damping mechanisms like "prediction steps" to prevent training from spiraling (Stabilizing Adversarial Nets with Prediction Methods).
  • Adaptive Optimizers: Methods like Adam and the newer MADGRAD by Aaron Defazio and Samy Jelassi attempt to bridge the gap by using adaptive preconditioning. While their formal proofs often still lean on convex assumptions, their success comes from their ability to navigate the high-dimensional geometry of non-convex space more effectively than simple SGD.
  • Large Language Models: As we scale to LLMs, the "mid-flight" adjustment of learning rates becomes critical. Shreyas Subramanian and colleagues demonstrated in Hop, skip, jump to Convergence that even in highly non-convex landscapes, loss trajectories for different learning rates eventually contract toward one another, providing a theoretical justification for why we can "jump" between convergence paths during training.

The research direction is clear: the field is moving away from trying to force deep learning into a convex box. Instead, researchers are building a new "non-convex calculus" that treats the complexity, jaggedness, and high-dimensionality of neural networks as primary features rather than obstacles to be assumed away.

Papers in this version

Showing 60 of 1,084 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reconciling_optimization_theory_with_non_25a9321b,
  title = {Reconciling Optimization Theory with Non-Convex Loss Landscapes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconciling-optimization-theory-with-non-convex-loss-landscapes/rdr_25a9321b8dd34981ae17e41fe7799709},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_reconciling_optimization_theory_with_non_25a9321b_v1,
  title = {Reconciling Optimization Theory with Non-Convex Loss Landscapes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconciling-optimization-theory-with-non-convex-loss-landscapes/rdr_25a9321b8dd34981ae17e41fe7799709/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}