In classical optimization theory, we often assume a "global speed limit" for how fast gradients can change. This property, known as Lipschitz smoothness, assumes that the curvature of the loss landscape—the Hessian—is bounded by a single constant $L$ everywhere. If you know $L$, you can safely pick a step size (like $1/L$) and guarantee that your optimizer won't fly off the rails.
However, modern deep learning practitioners have long known that this "flat earth" assumption is false. Models like LSTMs and Transformers inhabit "spiky" landscapes where the curvature isn't constant. Instead, the landscape is often sharpest exactly where the gradients are largest. This empirical reality explains why "tricks" like gradient clipping and learning rate warmup are essential for stability, yet these techniques remained theoretically mysterious for decades.
A new research direction, often termed generalized smoothness or $(L_0, L_1)$-smoothness, bridges this gap. By modeling curvature as a function of the gradient itself, researchers are finally providing a rigorous justification for the most successful heuristics in deep learning.
The Failure of Constant Smoothness
Traditional proofs for Stochastic Gradient Descent (SGD) rely on the assumption that $\|\nabla^2 f(x)\| \leq L$. But as Jingzhao Zhang and Tianxing He point out in Why gradient clipping accelerates training: A theoretical justification for adaptivity, this fails to capture the behavior of complex neural networks. In practice, regions with large gradients are often the most unstable. If the curvature grows as the gradient grows, a fixed step size that works in a flat valley will cause the model to diverge the moment it hits a steep hill.
To fix this, researchers like Daniil Vankov and Sebastian U. Stich have popularized the $(L_0, L_1)$-smoothness framework. In this model, the Hessian norm is bounded by an affine function of the gradient norm: $$\|\nabla^2 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$$ This simple change—allowing the "speed limit" to scale with the gradient—allows theory to finally match the "wild" landscapes of modern AI.
Why Gradient Clipping Works
Gradient clipping is the practice of capping the gradient norm at a threshold. While it seems like a crude hack, the generalized smoothness framework reveals it is actually a mathematically principled way to handle $(L_0, L_1)$ landscapes.
When the curvature increases with the gradient, a large gradient is a signal that the landscape is becoming dangerously sharp. By clipping the gradient, we effectively reduce the step size in these volatile regions. Aleksandr Beznosikov and Eduard Gorbunov demonstrate in Convergence of Clipped-SGD for Convex (L0,L1)-Smooth Optimization with Heavy-Tailed Noise that clipping is a robust, "all-in-one" solution that stabilizes training even when the noise in the data is heavy-tailed and unpredictable.
The Secret to Adaptive Optimizers
Adaptive methods like Adam and AdaGrad are the workhorses of Large Language Model (LLM) training. For years, theory suggested they shouldn't be much better than SGD in non-convex settings. However, Yushun Zhang and Tie-Yan Liu showed in Provable Adaptivity of Adam under Non-uniform Smoothness that Adam’s true strength lies in its ability to handle non-uniform smoothness.
Because Adam scales its updates based on the moving average of past gradients, it naturally adapts to the local "curviness" of the landscape. Recent work by Yuxing Liu and Tong Zhang in AdaGrad under Anisotropic Smoothness further refines this, showing that modern models exhibit anisotropic smoothness—where the curvature varies significantly across different parameter dimensions. Adaptive methods exploit this by taking larger steps in flat directions and smaller steps in sharp ones.
Justifying Warmup and Line-Search
The framework also sheds light on other common training strategies:
- Learning Rate Warmup: Why start with a tiny learning rate? Foivos Alimisis and colleagues argue in Why Do We Need Warm-up? A Theoretical Perspective that at the start of training, the model is often in a high-loss, high-curvature region. Warmup prevents the optimizer from taking a massive, destabilizing step before it reaches a smoother part of the landscape.
- Line-Search: While often considered too expensive for deep learning, Sharan Vaswani and Reza Babanezhad show in Armijo Line-search Makes (Stochastic) Gradient Descent Go Fast that for non-uniformly smooth functions, adaptive step sizes can achieve fundamentally faster convergence rates (linear vs. sublinear) than fixed schedules.
A Unified View of Training Stability
By moving away from global constants and toward gradient-dependent models, the field is developing a unified theory of training stability. Whether it is Dmitry Kovalev proving the acceleration of adaptive preconditioning in SGD with Adaptive Preconditioning or Thomas Pethick extending these ideas to non-Euclidean settings in Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothness, the message is clear: the most effective deep learning optimizers are those that respect the relationship between a landscape's slope and its curviness.