Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness

Vien V. Mai, Mikael Johansson

Abstract

Gradient clipping is a ubiquitous technique in deep learning used to mitigate the exploding gradient problem. Despite its practical success, theoretical justifications for clipping have historically relied on restrictive assumptions such as Lipschitz continuity or bounded gradients. This paper provides a rigorous theoretical framework for Clipped Stochastic Gradient Descent (SGD) and its momentum-based variants. We demonstrate that clipping serves as a fundamental stability anchor, enabling convergence on functions with arbitrary growth where standard SGD fails. Furthermore, we establish that clipped methods achieve optimal convergence rates for weakly convex functions, bridging the gap between heuristic practice and optimization theory.

Introduction

In classical optimization theory, the convergence of Stochastic Gradient Descent (SGD) is typically predicated on the assumption that the objective function is $L$-smooth (Lipschitz continuous gradients) or has a bounded gradient. However, modern machine learning architectures often involve "steep" landscapes where gradients grow faster than quadratically. In these scenarios, standard SGD is prone to numerical instability; a single large gradient update can propel the iterates toward infinity, a phenomenon known as the exploding gradient problem.

To combat this, practitioners employ gradient clipping—capping the gradient magnitude at a threshold $\gamma$. While effective, the theoretical implications of this "hack" have remained under-explored, particularly regarding how the bias introduced by clipping affects convergence rates and whether it truly stabilizes the algorithm in a mathematical sense. This paper addresses these questions by analyzing Clipped SGD and the Clipped Stochastic Heavy Ball (SHB) method.

Methodology

Algorithm Framework

The primary focus is on Clipped SGD, where the update rule is modified such that the stochastic gradient $g_t$ is projected onto a ball of radius $\gamma$: $$x_{t+1} = x_t - \eta \cdot \text{clip}(g_t, \gamma)$$ where $\text{clip}(g, \gamma) = \min(1, \gamma/\|g\|)g$. The authors further extend this analysis to the Clipped Stochastic Heavy Ball (SHB) method, incorporating a momentum term to accelerate convergence while maintaining the clipping safeguard.

Theoretical Approach

The research employs three distinct analytical lenses:

  1. Stability Analysis: The authors prove that clipped iterates are bounded by the sum of the stepsizes, preventing the super-exponential divergence seen in vanilla SGD when encountering steep regions.
  2. Growth Modeling: The algorithms are tested against functions exhibiting "arbitrary growth" (e.g., exponential) and "polynomial growth."
  3. Lyapunov Analysis: For weakly convex functions—a class that encompasses many neural network landscapes—the authors construct a novel Lyapunov function. This allows them to prove that the directional bias introduced by clipping does not impede the algorithm's ability to reach optimal convergence points.

Empirical Evaluation

The theoretical findings were validated across three diverse experimental setups designed to test the limits of optimization stability:

  • Phase Retrieval: A non-convex, non-smooth problem characterized by rapidly growing gradients. This served as a stress test for stability.
  • Absolute Linear Regression: A convex but non-smooth task used to determine if clipping introduces unnecessary overhead or performance degradation on simpler objectives.
  • Image Classification (CIFAR-10): A practical deep learning application using the ResNet-18 architecture to evaluate the impact of clipping on generalization and test accuracy.

Key Insights

Clipping as a Stability Anchor

The most significant finding is that vanilla SGD can diverge super-exponentially on steep functions, even with infinitesimal stepsizes. Clipping acts as a necessary stabilizer, ensuring the algorithm remains within a "searchable" region of the parameter space. This transforms clipping from a heuristic trick into a fundamental requirement for optimizing non-Lipschitz functions.

Optimal Rates for Complex Functions

For functions with polynomial growth, the authors demonstrate that Clipped SGD achieves a convergence rate of $O(1/\sqrt{K})$. This matches the optimal rates for Lipschitz-continuous functions, suggesting that clipping provides robustness against steepness without sacrificing theoretical efficiency.

Resolving Momentum Bias

Clipping inherently changes the gradient's direction, which traditionally complicates momentum-based proofs. The authors resolve this by showing that if the clipping threshold is set appropriately (e.g., $\gamma \geq 2L$), the update direction remains aligned with the "Moreau envelope"—a smoothed approximation of the objective function—ensuring consistent progress toward the minimum.

Results

  • Stability Gains: In Phase Retrieval tasks, vanilla SGD and SHB diverged at most learning rates. In contrast, their clipped counterparts remained stable and converged across a broad range of initial stepsizes (from $10^{-2}$ to $10^{2}$).
  • Accuracy and Robustness: On the CIFAR-10 dataset, Clipped SGD and SHB achieved superior test accuracies (approximately 95%) compared to Adam. Furthermore, clipped methods demonstrated significantly higher robustness to hyperparameter tuning, maintaining high performance across a wider range of learning rates.
  • Theoretical Convergence: The paper provides the first proof of a $O(1/\sqrt{K})$ convergence rate for weakly convex problems using clipped methods, establishing a new standard for robust optimization theory.

Conclusion and Future Work

This research provides the "theoretical soul" for gradient clipping, proving it is a mathematically sound method for stabilizing optimization in complex landscapes. By moving beyond the limitations of Lipschitz continuity, the authors provide a framework that better reflects the realities of training modern deep learning models.

Future research directions include the development of fully adaptive clipping thresholds that can automatically tune $\gamma$ during training while preserving the strong stability guarantees established here. Additionally, extending these proofs to more general non-convex landscapes without specific growth assumptions remains an open challenge.

Summary updated

Abstract

Stochastic gradient algorithms are often unstable when applied to functions that do not have Lipschitz-continuous and/or bounded gradients. Gradient clipping is a simple and effective technique to stabilize the training process for problems that are prone to the exploding gradient problem. Despite its widespread popularity, the convergence properties of the gradient clipping heuristic are poorly understood, especially for stochastic problems. This paper establishes both qualitative and quantitative convergence results of the clipped stochastic (sub)gradient method (SGD) for non-smooth convex functions with rapidly growing subgradients. Our analyses show that clipping enhances the stability of SGD and that the clipped SGD algorithm enjoys finite convergence rates in many cases. We also study the convergence of a clipped method with momentum, which includes clipped SGD as a special case, for weakly convex problems under standard assumptions. With a novel Lyapunov analysis, we show that the proposed method achieves the best-known rate for the considered class of problems, demonstrating the effectiveness of clipped methods also in this regime. Numerical results confirm our theoretical developments.
BibTeX
@inproceedings{mai2021stability,
  author = {Vien V. Mai and Mikael Johansson},
  title = {{Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure