Abstract
Gradient clipping is a ubiquitous technique in deep learning used to mitigate the exploding gradient problem. Despite its practical success, theoretical justifications for clipping have historically relied on restrictive assumptions such as Lipschitz continuity or bounded gradients. This paper provides a rigorous theoretical framework for Clipped Stochastic Gradient Descent (SGD) and its momentum-based variants. We demonstrate that clipping serves as a fundamental stability anchor, enabling convergence on functions with arbitrary growth where standard SGD fails. Furthermore, we establish that clipped methods achieve optimal convergence rates for weakly convex functions, bridging the gap between heuristic practice and optimization theory.
Introduction
In classical optimization theory, the convergence of Stochastic Gradient Descent (SGD) is typically predicated on the assumption that the objective function is $L$-smooth (Lipschitz continuous gradients) or has a bounded gradient. However, modern machine learning architectures often involve "steep" landscapes where gradients grow faster than quadratically. In these scenarios, standard SGD is prone to numerical instability; a single large gradient update can propel the iterates toward infinity, a phenomenon known as the exploding gradient problem.
To combat this, practitioners employ gradient clipping—capping the gradient magnitude at a threshold $\gamma$. While effective, the theoretical implications of this "hack" have remained under-explored, particularly regarding how the bias introduced by clipping affects convergence rates and whether it truly stabilizes the algorithm in a mathematical sense. This paper addresses these questions by analyzing Clipped SGD and the Clipped Stochastic Heavy Ball (SHB) method.
Methodology
Algorithm Framework
The primary focus is on Clipped SGD, where the update rule is modified such that the stochastic gradient $g_t$ is projected onto a ball of radius $\gamma$: $$x_{t+1} = x_t - \eta \cdot \text{clip}(g_t, \gamma)$$ where $\text{clip}(g, \gamma) = \min(1, \gamma/\|g\|)g$. The authors further extend this analysis to the Clipped Stochastic Heavy Ball (SHB) method, incorporating a momentum term to accelerate convergence while maintaining the clipping safeguard.
Theoretical Approach
The research employs three distinct analytical lenses:
- Stability Analysis: The authors prove that clipped iterates are bounded by the sum of the stepsizes, preventing the super-exponential divergence seen in vanilla SGD when encountering steep regions.
- Growth Modeling: The algorithms are tested against functions exhibiting "arbitrary growth" (e.g., exponential) and "polynomial growth."
- Lyapunov Analysis: For weakly convex functions—a class that encompasses many neural network landscapes—the authors construct a novel Lyapunov function. This allows them to prove that the directional bias introduced by clipping does not impede the algorithm's ability to reach optimal convergence points.
Empirical Evaluation
The theoretical findings were validated across three diverse experimental setups designed to test the limits of optimization stability:
- Phase Retrieval: A non-convex, non-smooth problem characterized by rapidly growing gradients. This served as a stress test for stability.
- Absolute Linear Regression: A convex but non-smooth task used to determine if clipping introduces unnecessary overhead or performance degradation on simpler objectives.
- Image Classification (CIFAR-10): A practical deep learning application using the ResNet-18 architecture to evaluate the impact of clipping on generalization and test accuracy.
Key Insights
Clipping as a Stability Anchor
The most significant finding is that vanilla SGD can diverge super-exponentially on steep functions, even with infinitesimal stepsizes. Clipping acts as a necessary stabilizer, ensuring the algorithm remains within a "searchable" region of the parameter space. This transforms clipping from a heuristic trick into a fundamental requirement for optimizing non-Lipschitz functions.
Optimal Rates for Complex Functions
For functions with polynomial growth, the authors demonstrate that Clipped SGD achieves a convergence rate of $O(1/\sqrt{K})$. This matches the optimal rates for Lipschitz-continuous functions, suggesting that clipping provides robustness against steepness without sacrificing theoretical efficiency.
Resolving Momentum Bias
Clipping inherently changes the gradient's direction, which traditionally complicates momentum-based proofs. The authors resolve this by showing that if the clipping threshold is set appropriately (e.g., $\gamma \geq 2L$), the update direction remains aligned with the "Moreau envelope"—a smoothed approximation of the objective function—ensuring consistent progress toward the minimum.
Results
- Stability Gains: In Phase Retrieval tasks, vanilla SGD and SHB diverged at most learning rates. In contrast, their clipped counterparts remained stable and converged across a broad range of initial stepsizes (from $10^{-2}$ to $10^{2}$).
- Accuracy and Robustness: On the CIFAR-10 dataset, Clipped SGD and SHB achieved superior test accuracies (approximately 95%) compared to Adam. Furthermore, clipped methods demonstrated significantly higher robustness to hyperparameter tuning, maintaining high performance across a wider range of learning rates.
- Theoretical Convergence: The paper provides the first proof of a $O(1/\sqrt{K})$ convergence rate for weakly convex problems using clipped methods, establishing a new standard for robust optimization theory.
Conclusion and Future Work
This research provides the "theoretical soul" for gradient clipping, proving it is a mathematically sound method for stabilizing optimization in complex landscapes. By moving beyond the limitations of Lipschitz continuity, the authors provide a framework that better reflects the realities of training modern deep learning models.
Future research directions include the development of fully adaptive clipping thresholds that can automatically tune $\gamma$ during training while preserving the strong stability guarantees established here. Additionally, extending these proofs to more general non-convex landscapes without specific growth assumptions remains an open challenge.
Summary updated