Combining learning rate decay and weight decay with complexity gradient descent - Part I

Pierre H. Richemond, Yike GuoarXiv 2019 · OpenReview

Abstract

The role of $L^2$ regularization, in the specific case of deep neural networks rather than more traditional machine learning models, is still not fully elucidated. We hypothesize that this complex interplay is due to the combination of overparameterization and high dimensional phenomena that take place during training and make it unamenable to standard convex optimization methods. Using insights from statistical physics and random fields theory, we introduce a parameter factoring in both the level of the loss function and its remaining nonconvexity: the complexity. We proceed to show that it is desirable to proceed with complexity gradient descent. We then show how to use this intuition to derive novel and efficient annealing schemes for the strength of $L^2$ regularization when performing standard stochastic gradient descent in deep neural networks.

Introduction

Despite the ubiquitous use of weight decay and learning rate annealing in deep learning, a rigorous theoretical understanding of why these techniques work in high-dimensional, non-convex landscapes remains elusive. Standard optimization theory often relies on convexity, which is absent in deep neural networks. This paper bridges this gap by applying tools from statistical physics—specifically the theory of p-spin spherical spin glasses—to model the neural loss landscape as a complex system. We propose that navigating the geometry of this landscape via a "complexity" objective is key to finding minima that generalize well.

Theoretical Framework: The Geometry of Complexity

To quantify the landscape's difficulty, we utilize the Kac-Rice formula to count critical points (minima, maxima, and saddle points). We define Complexity ($Σ$) as the log-count of these points at a given energy (loss) level $ε$. A central variable in this analysis is the index $α$, representing the fraction of negative eigenvalues in the Hessian.

The relationship between the index $\alpha$ and the most probable energy level $\epsilon$.

As illustrated in Figure 1, the index $α$ varies with the energy level. At high energies, the landscape is dominated by saddle points (high $α$), while lower energy levels exhibit a higher density of stable minima. Complexity Gradient Descent aims to follow the path that minimizes this index most efficiently.

The Symmetry of Loss and Curvature

One of the paper's most significant insights is the mathematical symmetry between lowering the loss (energy $ε$) and adjusting the average curvature (average eigenvalue $\bar{\lambda}$). Weight decay ($L^2$ regularization) acts as a landscape shifter, effectively "folding" the negative spectrum of the Hessian back into the positive, thereby converting saddle points into minima.

The index $\alpha$ as a function of the rescaled average eigenvalue $\bar{\lambda}$.

Figure 4 shows that increasing the average eigenvalue (via weight decay) reduces the index $α$, mirroring the effect of reducing energy. This suggests that weight decay and learning rate decay are two different ways to move through the same complexity space. This symmetry allows us to link the two hyperparameters through a simple differential equation: $|dε / ε| = |dλ / λ|$.

Complexity Gradient Descent and Annealing Schemes

We propose Complexity Gradient Descent, an optimization framework that follows the gradient of the complexity function $Σ$ rather than just the loss. This approach inherently prioritizes "flat" minima, which are associated with better generalization. From this theory, we derive three novel annealing schemes for weight decay ($λ$):

  • Matched Annealing: Decaying $λ$ in strict proportion to loss decay.
  • Cosine Annealing: A periodic schedule for $λ$ that allows the optimizer to escape local traps and explore flatter regions.
  • Two-step Annealing: An iterative approach alternating between loss minimization and curvature optimization.

Limitations and Future Work

While the framework is mathematically rigorous, it relies on the assumption that neural network initializations behave like Gaussian random fields. Furthermore, as this is "Part I," the focus is entirely theoretical. Empirical validation on standard benchmarks like ImageNet and CIFAR is not discussed here but will be the subject of Part II, where the practical performance of these complexity-based annealing schemes will be evaluated against standard optimizers like ADAM.

Summary updated

Abstract

The role of $L^2$ regularization, in the specific case of deep neural networks rather than more traditional machine learning models, is still not fully elucidated. We hypothesize that this complex interplay is due to the combination of overparameterization and high dimensional phenomena that take place during training and make it unamenable to standard convex optimization methods. Using insights from statistical physics and random fields theory, we introduce a parameter factoring in both the level of the loss function and its remaining nonconvexity: the \emph{complexity}. We proceed to show that it is desirable to proceed with \emph{complexity gradient descent}. We then show how to use this intuition to derive novel and efficient annealing schemes for the strength of $L^2$ regularization when performing standard stochastic gradient descent in deep neural networks.
BibTeX
@misc{richemond2019combining,
  author = {Pierre H. Richemond and Yike Guo},
  title = {{Combining learning rate decay and weight decay with complexity gradient descent - Part I}},
  year = {2019},
  eprint = {1902.02881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.02881},
}
Figure