Step by Step: Adaptive Gradient Descent for Training L-Lipschitz Neural Networks

K. Sung, Anastasis Kratsios, Noah Forman, Samu, Steven, Kratsios, Anastasis

Abstract: We demonstrate that applying an eventual decay to the learning rate (LR) in empirical risk minimization (ERM), where the mean-squared-error loss is minimized using standard gradient descent (GD) for training a two-layer neural network with Lipschitz activation functions, ensures that the resulting network exhibits a high degree of Lipschitz regularity, that is, a small Lipschitz constant. Moreover, we show that this decay does not hinder the convergence rate of the empirical risk, now measured with the Huber loss, toward a critical point of the non-convex empirical risk. From these findings, we derive generalization bounds for two-layer neural networks trained with GD and a decaying LR with a sub-linear dependence on its number of trainable parameters, suggesting that the statistical behaviour of these networks is independent of overparameterization. We validate our theoretical results with a series of toy numerical experiments, where surprisingly, we observe that networks trained with constant step size GD exhibit similar learning and regularity properties to those trained with a decaying LR. This suggests that neural networks trained with standard GD may already be highly regular learners.

Introduction

In deep learning, achieving Lipschitz regularity—where small changes in input lead to proportional changes in output—is a primary goal for model robustness and generalization. High Lipschitz regularity is often considered the "holy grail" for making models robust to noise and is a requirement for advanced techniques like Wasserstein GANs. Traditionally, this is achieved through heavy mathematical penalties or architectural constraints that can limit model expressivity. This paper explores a simpler alternative: achieving a target Lipschitz constant solely through adaptive learning rate (LR) adjustments during standard gradient descent.

Methodology: Adaptive Learning Rates and Huber Loss

The core contribution of this work is the definition of "GD LR Decay Conditions." Rather than using a fixed learning rate, the rate is adjusted based on the current gradient and a specific rate function, such as exponential or polynomial decay.

The Hybrid Strategy

The researchers propose a "Hybrid" strategy to balance training speed and regularity. As illustrated in the figure below, training can begin with a constant learning rate to rapidly approach a solution, then switch to the prescribed decay at a specific iteration $T$ to "lock in" the desired Lipschitz regularity.

Learning rate decay strategies: The hybrid approach starts with a constant rate and transitions to a decay condition to ensure Lipschitz regularity.

Bridging MSE and Huber Loss

While the networks are trained using standard Mean Squared Error (MSE), the theoretical analysis of convergence is conducted using the Huber loss. The Huber loss acts as a 1-Lipschitz bridge, allowing the authors to prove convergence to a critical point of the non-convex empirical risk without sacrificing the benefits of MSE-based training. This approach effectively bridges the gap between optimization theory and statistical stability.

Huber Loss vs MSE: The Huber loss provides a robust, Lipschitz-continuous alternative for analyzing convergence in empirical risk minimization.

Experimental Validation

The theory was tested on synthetic benchmarks common in non-parametric regression, specifically the Forrester and Franke functions. These functions allow for rigorous evaluation of how models handle varying oscillations and complex shapes. Ablation studies across varying network widths (up to 128 neurons), depths (up to 10 layers), and sample sizes (up to 10,000 points) were conducted to track the evolution of the Lipschitz constant under different training regimes.

Key Insights and Results

  • Guaranteed Regularity and Convergence: Theorem 3.4 provides a high-probability bound on the Lipschitz constant, while Theorem 3.10 proves that LR decay does not hinder convergence, maintaining an optimal rate of $O(1/\sqrt{T})$.
  • Generalization and Overparameterization: The derived generalization bounds depend sub-linearly on network width, supporting the idea that overparameterization does not necessarily degrade performance if training is appropriately regularized via the learning rate.
  • The "Constant LR" Surprise: Experiments revealed that even constant learning rate GD often results in highly regular models. This suggests that standard Gradient Descent might have an inherent "hidden" bias toward learning smooth, regular functions.

Conclusion and Future Directions

While these results provide a strong foundation for Multilayer Perceptrons (MLPs), future work is needed to extend these findings to modern architectures like Transformers and CNNs. Additionally, exploring the applicability of these decay conditions to momentum-based optimizers like Adam and scaling to high-dimensional, real-world datasets remains an open area for research.

Summary updated

Abstract

We demonstrate that applying an eventual decay to the learning rate (LR) in empirical risk minimization (ERM), where the mean-squared-error loss is minimized using standard gradient descent (GD) for training a two-layer neural network with Lipschitz activation functions, ensures that the resulting network exhibits a high degree of Lipschitz regularity, that is, a small Lipschitz constant. Moreover, we show that this decay does not hinder the convergence rate of the empirical risk, now measured with the Huber loss, toward a critical point of the non-convex empirical risk. From these findings, we derive generalization bounds for two-layer neural networks trained with GD and a decaying LR with a sub-linear dependence on its number of trainable parameters, suggesting that the statistical behaviour of these networks is independent of overparameterization. We validate our theoretical results with a series of toy numerical experiments, where surprisingly, we observe that networks trained with constant step size GD exhibit similar learning and regularity properties to those trained with a decaying LR. This suggests that neural networks trained with standard GD may already be highly regular learners.
BibTeX
@misc{sung2025step,
  author = {K. Sung and Anastasis Kratsios and Noah Forman and Samu, Steven and Kratsios, Anastasis},
  title = {{Step by Step: Adaptive Gradient Descent for Training L-Lipschitz Neural Networks}},
  year = {2025},
  eprint = {2502.03792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.03792},
}
Figure