Three Mechanisms of Weight Decay Regularization

Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse

TL;DR

Weight decay is a staple of the deep learning toolbox, yet its success is often misunderstood as simple $L_2$ regularization. This paper reveals that weight decay actually operates through three distinct, hidden mechanisms—boosting effective learning rates, regularizing Jacobian norms, and managing optimizer damping—depending on your choice of optimizer and architecture.

What Problem Does This Paper Address?

For decades, the standard narrative has been that weight decay (WD) is just $L_2$ regularization in disguise: by penalizing large weights, we prevent overfitting. However, recent deep learning developments have poked holes in this theory. For instance, weight decay is highly effective in networks using Batch Normalization (BN), even though BN makes the network's output invariant to the scale of the weights. Furthermore, in adaptive optimizers like Adam, "literal" weight decay (scaling weights down every step) performs significantly better than adding an $L_2$ penalty to the loss function.

The authors set out to demystify why weight decay works so well across different settings, moving beyond the simplistic $L_2$ interpretation to find the actual drivers of generalization.

Methods

The researchers conducted an extensive empirical study comparing weight decay and $L_2$ regularization across three major optimization algorithms:

  1. SGD: The workhorse of deep learning.
  2. Adam: The go-to adaptive optimizer.
  3. K-FAC: A sophisticated second-order optimizer (Kronecker-Factored Approximate Curvature).

They tested these optimizers on various architectures (VGG16, ResNet32) with and without Batch Normalization. By isolating specific layers for weight decay and mathematically analyzing the update rules, they identified three distinct regularizing behaviors.

Datasets

The study primarily utilized standard computer vision benchmarks:

  • CIFAR-10
  • CIFAR-100
  • MNIST (specifically for analyzing the relationship between K-FAC and Jacobian norms)

Key Insights and Ideas

The "aha" moment of the paper is that weight decay isn't doing one thing; it’s doing three different things depending on the context:

  • Mechanism I: The Effective Learning Rate Boost (SGD/Adam + BN). In networks with Batch Normalization, the scale of the weights doesn't change the output, but it does change the gradient. Smaller weights lead to larger gradients relative to the weight norm. This effectively increases the "effective learning rate," which introduces more gradient noise. This noise acts as a stochastic regularizer, helping the model find flatter, more generalizable minima.
  • Mechanism II: Jacobian Regularization (K-FAC without BN). For second-order optimizers like K-FAC, the authors proved that weight decay is mathematically equivalent to regularizing the input-output Jacobian norm. Essentially, it forces the network to be less sensitive to small changes in the input, a known recipe for better generalization.
  • Mechanism III: Damping Control (K-FAC + BN). In practice, second-order optimizers use "damping" (adding a small constant to the curvature matrix) to stay stable. Without weight decay, weights grow large, causing the curvature to shrink and the fixed damping term to dominate. This effectively turns a smart second-order optimizer back into a simple first-order one. Weight decay keeps weights small, ensuring the optimizer retains its second-order "intelligence."

Key Results

The paper provides strong evidence that weight decay is superior to standard $L_2$ regularization in almost every modern setting.

  • Performance Boost: As seen in Figure 1, weight decay consistently outperforms both the baseline and $L_2$ regularization across all optimizers. Figure 1: Comparison of test accuracy across different optimizers and settings. Weight decay (dark green) consistently leads to the best performance.
Figure 1: Comparison of test accuracy of the networks trained with different optimizers on both CIFAR10 and CIFAR100. We compare Weight Decay regularization to L2 regularization and the Baseline (which used neither). Here, BN+Aug denotes the use of BN and data augmentation. K-FAC-G and K-FAC-F denote K-FAC using Gauss-Newton and Fisher matrices as the preconditioner, respectively. The results suggest that weight decay leads to improved performance across different optimizers and settings.
  • Closing the Gap: Weight decay was found to close the generalization gap between first-order (SGD) and second-order (K-FAC) optimizers, as well as the notorious gap between small-batch and large-batch training.
  • Layer-wise Impact: In BN networks, the authors found that applying weight decay to the convolutional layers (which use BN) was responsible for almost all the generalization benefits, even though it shouldn't "matter" theoretically. Figure 2: Test accuracy over time. Applying weight decay to all layers (blue) or just convolutional layers (green) yields similar gains, while applying it only to the final fully-connected layer (cyan) barely helps.
Figure 2: Test accuracy as a function of training epoch for SGD and Adam on CIFAR-100 with different weight decay regularization schemes. baseline is the model without weight decay; wd-conv is the model with weight decay applied to all convolutional layers; wd-all is the model with weight decay applied to all layers; wd-fc is the model with weight decay applied to the last layer (fc). Most of the generalization effect of weight decay is due to applying it to layers with BN.

Limitations and Future Work

While the paper provides a much clearer picture of weight decay, it has some bounds:

  • Linearity Assumptions: The proof for Jacobian regularization (Mechanism II) is strictly derived for linear networks, though the authors provide empirical evidence that the correlation holds for non-linear deep networks.
  • Hyperparameter Sensitivity: The interactions between weight decay, learning rates, and damping are complex. The authors note that this complexity highlights a need for more automated hyperparameter tuning methods that can navigate these hidden dynamics without manual trial-and-error.
  • Potential for More Mechanisms: The authors acknowledge that there may be even more mechanisms at play that haven't been identified yet.

Summary updated

Abstract

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.
BibTeX
@inproceedings{zhang2019three,
  author = {Guodong Zhang and Chaoqi Wang and Bowen Xu and Roger Grosse},
  title = {{Three Mechanisms of Weight Decay Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=epw2WHiTge},
}
Figure