The mathematical foundation of deep learning is built on a paradox. Most of the theoretical guarantees we rely on—convergence rates, stability bounds, and optimal learning rates—are derived from the world of convex optimization. In a convex world, every local minimum is a global minimum, and the "landscape" of the loss function is a smooth, predictable bowl.
However, modern neural networks operate in a "messy" reality. As Hoang Tran and Qinzi Zhang point out in Reevaluating Theoretical Analysis Methods for Optimization in Deep Learning, the loss landscapes of deep networks are high-dimensional, non-convex, and often jagged. This creates a significant gap: our optimizers work brilliantly in practice, but the theories explaining why they work often rely on assumptions that are demonstrably false.
The Failure of Classical Assumptions
In classical theory, two pillars support most proofs: convexity (the function curves upward everywhere) and smoothness (the gradient doesn't change too abruptly). When these hold, we can guarantee that an algorithm like Gradient Descent will find the bottom of the bowl.
In deep learning, these pillars crumble:
- Non-Convexity: Neural networks are riddled with saddle points and local minima. Krishna Reddy Kesari proved that finding the global minimum of a general non-convex function is an exponential-time problem—a "needle-in-a-haystack" scenario that should, theoretically, make deep learning impossible (First Order Methods take Exponential Time to Converge to Global Minimizers of Non-Convex Functions).
- Jaggedness: While theory often assumes smoothness, real-world training involves "sharp" regions. Hoang Tran found that while smoothness-based theories largely fail to describe practice, convexity-based math surprisingly holds up better than expected, acting as a "good enough" approximation for why models converge.
Bridging the Gap: New Theoretical Frameworks
To move past these restrictive assumptions, researchers are developing new ways to characterize the non-convex journey of an optimizer.
From Points to Manifolds
Instead of looking for a single "optimal point," Benjamin Fehrman, Arnulf Jentzen, and Benjamin Gess suggest we should view convergence as a movement toward manifolds of minima. In Convergence rates for the stochastic gradient descent method for non-convex objective functions, they provide rigorous proofs for SGD that don't require global convexity, better reflecting how models actually settle into stable regions.
The Power of Expressivity
In the realm of meta-learning, the non-convexity problem is even more acute because we are optimizing the optimization process itself. Lingxiao Wang and Zhaoran Wang showed in On the Global Optimality of Model-Agnostic Meta-Learning that if a neural network is "sufficiently expressive," the stationary points found by MAML are actually approximately globally optimal. This suggests that the architecture's capacity can sometimes "overcome" the theoretical difficulties of the non-convex landscape.
Discrete Jumps vs. Continuous Flow
Traditional theory often models gradient descent as a smooth, continuous "flow." However, Amirkeivan Mohtashami argues in Special Properties of Gradient Descent with Large Learning Rates that the discreteness of our steps is a feature, not a bug. Large learning rates allow the optimizer to physically "jump" over sharp local minima that a continuous flow would get stuck in. This "escaping mechanism" is a purely non-convex phenomenon that convex theory completely misses.
Implications for Robustness and Scaling
This theoretical disconnect isn't just an academic concern; it affects how we build robust and efficient systems.
- Adversarial Training: Standard proofs for stabilizing GANs or adversarial training often assume "convex-concave" surfaces. Abhay Kumar Yadav and Sohil Shah found that these guarantees fail in deep learning, necessitating empirical damping mechanisms like "prediction steps" to prevent training from spiraling (Stabilizing Adversarial Nets with Prediction Methods).
- Adaptive Optimizers: Methods like Adam and the newer MADGRAD by Aaron Defazio and Samy Jelassi attempt to bridge the gap by using adaptive preconditioning. While their formal proofs often still lean on convex assumptions, their success comes from their ability to navigate the high-dimensional geometry of non-convex space more effectively than simple SGD.
- Large Language Models: As we scale to LLMs, the "mid-flight" adjustment of learning rates becomes critical. Shreyas Subramanian and colleagues demonstrated in Hop, skip, jump to Convergence that even in highly non-convex landscapes, loss trajectories for different learning rates eventually contract toward one another, providing a theoretical justification for why we can "jump" between convergence paths during training.
The research direction is clear: the field is moving away from trying to force deep learning into a convex box. Instead, researchers are building a new "non-convex calculus" that treats the complexity, jaggedness, and high-dimensionality of neural networks as primary features rather than obstacles to be assumed away.