In classical optimization theory, a common "safety rail" used to prove that algorithms like Stochastic Gradient Descent (SGD) converge is the bounded gradient assumption. This assumption posits that the norm of the gradient $\|\nabla f(x)\|$ never exceeds some constant $G$ across the entire parameter space.
While mathematically convenient, this assumption is fundamentally at odds with reality. Even a simple quadratic function $f(x) = x^2$ has gradients that grow infinitely as $x$ moves away from the origin. In deep learning, where loss landscapes are complex and high-dimensional, assuming gradients are globally capped is often unrealistic. Recent research has focused on dismantling this assumption, providing tighter, more robust theories that align with how models actually behave in production.
The Gap Between Theory and Practice
The bounded gradient assumption is often used to control the "noise" in stochastic updates. If gradients are bounded, the variance of the stochastic estimates is also naturally constrained. However, as Quoc Tran-Dinh and Lam M. Nguyen point out in A Unified Convergence Analysis for Shuffling-Type Gradient Methods, this assumption often implies an unrealistic implicit constraint on the parameter space, effectively pretending the model stays within a small, well-behaved box.
When we remove these rails, we face a core challenge: if the gradient can grow arbitrarily large, a single noisy step could theoretically catapult the model parameters into a region where the gradient is even larger, leading to a catastrophic divergence. To prove this doesn't happen, researchers have developed several key mechanisms.
Self-Bounding Gradients and Affine Variance
One way to move past global bounds is to recognize that gradients often bound themselves relative to the objective value. Yunwen Lei and colleagues in Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions utilize a "self-bounding" property. Instead of a global constant $G$, they show that the gradient norm can be controlled by the function's smoothness and the current distance to the optimum.
Similarly, Amit Attia and Tomer Koren analyze SGD with AdaGrad Stepsizes under "affine variance." This condition assumes the noise variance scales with the magnitude of the gradient itself. By proving that adaptive methods naturally compensate for this scaling, they show that algorithms like AdaGrad are "fully adaptive"—they don't need to know the smoothness or noise levels beforehand to remain stable in the presence of unbounded gradients.
The Offset Trick and Lyapunov Stability
In decentralized and non-convex settings, tracking progress is harder because different agents might be wandering in different directions. Aleksandar Armacki employs an offset trick in High-probability Convergence Guarantees of Decentralized SGD. This mathematical technique adjusts the "potential function" used to measure progress, effectively creating a buffer that absorbs stochastic noise and local curvature without requiring a global gradient cap.
For distributed systems, Ying Sun 0003 and Gesualdo Scutari introduced the SONATA framework in Distributed Nonconvex Constrained Optimization over Time-Varying Digraphs. They use a Lyapunov-like function—a concept from control theory—to prove that even if gradients are unbounded and the network is unreliable, the system's "energy" still dissipates toward a stationary point.
Adaptivity as a Natural Stabilizer
Adaptive optimizers like Adam and AdaGrad are particularly well-suited for unbounded landscapes because they normalize updates by the historical gradient magnitude. Bohan Wang and others proved in Closing the gap between the upper bound and lower bound of Adam's iteration complexity that Adam reaches optimal convergence rates without universally bounded gradients. The intuition is that if the gradient explodes, the denominator in the Adam update also explodes, naturally "braking" the step size.
This stabilization is also seen in LightSAM by Li Shen, which applies adaptive learning rates to sharpness-aware minimization. By decoupling the update magnitude from the raw gradient norm, these methods remain robust even when the loss landscape becomes extremely steep.
From Bounded Sets to Coercivity
If we can't assume the gradients are bounded, can we at least assume the model parameters (iterates) stay within a bounded set? Often, the answer is no. Andre Milzarek and Xiao Li propose a KL-based Analysis Framework that replaces bounded iterate assumptions with coercivity.
A function is coercive if its value goes to infinity as the input goes to infinity. By using this property, researchers can prove that an algorithm will either converge to a sensible solution or the function value will blow up—and since optimization algorithms are designed to minimize the function, they are naturally incentivized to stay within a "level set" where the parameters are effectively bounded. This shift from global assumptions to local, function-specific properties like level-coercivity (Alexander Bodard and Emanuel Laude in The inexact power augmented Lagrangian method) allows for a much more realistic theoretical treatment of modern machine learning.