TL;DR
Weight decay is a staple of the deep learning toolbox, yet its success is often misunderstood as simple $L_2$ regularization. This paper reveals that weight decay actually operates through three distinct, hidden mechanisms—boosting effective learning rates, regularizing Jacobian norms, and managing optimizer damping—depending on your choice of optimizer and architecture.
What Problem Does This Paper Address?
For decades, the standard narrative has been that weight decay (WD) is just $L_2$ regularization in disguise: by penalizing large weights, we prevent overfitting. However, recent deep learning developments have poked holes in this theory. For instance, weight decay is highly effective in networks using Batch Normalization (BN), even though BN makes the network's output invariant to the scale of the weights. Furthermore, in adaptive optimizers like Adam, "literal" weight decay (scaling weights down every step) performs significantly better than adding an $L_2$ penalty to the loss function.
The authors set out to demystify why weight decay works so well across different settings, moving beyond the simplistic $L_2$ interpretation to find the actual drivers of generalization.
Methods
The researchers conducted an extensive empirical study comparing weight decay and $L_2$ regularization across three major optimization algorithms:
- SGD: The workhorse of deep learning.
- Adam: The go-to adaptive optimizer.
- K-FAC: A sophisticated second-order optimizer (Kronecker-Factored Approximate Curvature).
They tested these optimizers on various architectures (VGG16, ResNet32) with and without Batch Normalization. By isolating specific layers for weight decay and mathematically analyzing the update rules, they identified three distinct regularizing behaviors.
Datasets
The study primarily utilized standard computer vision benchmarks:
- CIFAR-10
- CIFAR-100
- MNIST (specifically for analyzing the relationship between K-FAC and Jacobian norms)
Key Insights and Ideas
The "aha" moment of the paper is that weight decay isn't doing one thing; it’s doing three different things depending on the context:
- Mechanism I: The Effective Learning Rate Boost (SGD/Adam + BN). In networks with Batch Normalization, the scale of the weights doesn't change the output, but it does change the gradient. Smaller weights lead to larger gradients relative to the weight norm. This effectively increases the "effective learning rate," which introduces more gradient noise. This noise acts as a stochastic regularizer, helping the model find flatter, more generalizable minima.
- Mechanism II: Jacobian Regularization (K-FAC without BN). For second-order optimizers like K-FAC, the authors proved that weight decay is mathematically equivalent to regularizing the input-output Jacobian norm. Essentially, it forces the network to be less sensitive to small changes in the input, a known recipe for better generalization.
- Mechanism III: Damping Control (K-FAC + BN). In practice, second-order optimizers use "damping" (adding a small constant to the curvature matrix) to stay stable. Without weight decay, weights grow large, causing the curvature to shrink and the fixed damping term to dominate. This effectively turns a smart second-order optimizer back into a simple first-order one. Weight decay keeps weights small, ensuring the optimizer retains its second-order "intelligence."
Key Results
The paper provides strong evidence that weight decay is superior to standard $L_2$ regularization in almost every modern setting.
- Performance Boost: As seen in Figure 1, weight decay consistently outperforms both the baseline and $L_2$ regularization across all optimizers. Figure 1: Comparison of test accuracy across different optimizers and settings. Weight decay (dark green) consistently leads to the best performance.
- Closing the Gap: Weight decay was found to close the generalization gap between first-order (SGD) and second-order (K-FAC) optimizers, as well as the notorious gap between small-batch and large-batch training.
- Layer-wise Impact: In BN networks, the authors found that applying weight decay to the convolutional layers (which use BN) was responsible for almost all the generalization benefits, even though it shouldn't "matter" theoretically. Figure 2: Test accuracy over time. Applying weight decay to all layers (blue) or just convolutional layers (green) yields similar gains, while applying it only to the final fully-connected layer (cyan) barely helps.
Limitations and Future Work
While the paper provides a much clearer picture of weight decay, it has some bounds:
- Linearity Assumptions: The proof for Jacobian regularization (Mechanism II) is strictly derived for linear networks, though the authors provide empirical evidence that the correlation holds for non-linear deep networks.
- Hyperparameter Sensitivity: The interactions between weight decay, learning rates, and damping are complex. The authors note that this complexity highlights a need for more automated hyperparameter tuning methods that can navigate these hidden dynamics without manual trial-and-error.
- Potential for More Mechanisms: The authors acknowledge that there may be even more mechanisms at play that haven't been identified yet.
Summary updated