Abstract
This research uncovers a fundamental scaling law in deep learning: the time required for an overparameterized network to reach peak performance is inversely proportional to the $L_2$ regularization coefficient ($\lambda$). By characterizing this relationship, the paper demonstrates that $L_2$ regularization primarily controls the speed of training dynamics rather than just preventing overfitting. These insights lead to a method for predicting optimal regularization strengths and the introduction of $AutoL_2$, a dynamic schedule that improves both training efficiency and final accuracy.
Introduction
In classical statistics, $L_2$ regularization (weight decay) is viewed through the lens of the bias-variance tradeoff. In the context of deep learning, however, we frequently employ massive, overparameterized models that theoretically should overfit but often generalize remarkably well. This discrepancy suggests that the role of weight decay in neural networks is more complex than traditional theory suggests. Practitioners typically find the "optimal" $\lambda$ through exhaustive hyperparameter sweeps, a process that is computationally expensive and lacks a theoretical foundation. This work establishes a predictable relationship between $\lambda$, the learning rate ($\eta$), and training time to make the optimization process more principled and efficient.
The $1/\lambda$ Scaling Law
Through extensive empirical sweeps across various architectures—including Wide ResNets, CNNs, and fully connected networks—the authors discovered a robust scaling rule. The number of training steps $t_*$ required to reach maximum test accuracy is inversely proportional to the product of the learning rate and the regularization coefficient: $t_* \approx c/(\eta\lambda)$.
This discovery implies that $L_2$ regularization acts as a "clock" for the training process. For a sufficiently small $\lambda$, if the training duration is scaled by $1/\lambda$, the final test accuracy remains approximately constant. This "performance independence" suggests that the primary function of $L_2$ in deep learning is to modulate the speed at which the model's internal representations (kernels) evolve.
Practical Applications: Optimal $\lambda$ Prediction and $AutoL_2$
The established scaling law enables significant improvements in training workflows through two primary applications:
Predicting Optimal Regularization
By performing a short "pilot" run with a high $\lambda$ value, the constant $c$ for a specific architecture and dataset can be determined. This constant then allows for the calculation of the optimal $\lambda$ for any given training budget $T$. In experiments on CIFAR-10, this predictive method achieved test accuracies within 0.4% of those found through exhaustive hyperparameter tuning, but at a fraction of the computational cost.
The $AutoL_2$ Schedule
The authors introduce $AutoL_2$, a dynamic regularization schedule that starts with a high $\lambda$ to accelerate initial learning and subsequently decays it by a factor of 10 whenever the training loss or error plateaus.
As illustrated in Figure 1c, $AutoL_2$ reaches higher accuracy significantly faster than a tuned constant $\lambda$ baseline. This approach reduces the need for precise initial tuning and adapts the regularization strength to the current state of the training dynamics.
Theoretical Insights: Kernel Decay
To explain these empirical findings, the study analyzed $k$-homogeneous networks in the infinite-width limit. Derivations of gradient flow equations show that $L_2$ regularization causes the Neural Tangent Kernel (NTK) to decay over time. In deep networks, this means the feature representation itself is being compressed or "eroded" by the regularization. This explains why performance eventually deteriorates if a model is trained for too long with a fixed $\lambda$—a behavior distinct from linear models where the kernel remains static. The degree of homogeneity $k$ directly influences this decay scale, with deeper networks generally converging faster.
Conclusion and Future Work
This work provides a new perspective on weight decay, framing it as a controller of training speed in overparameterized regimes. While these scaling laws are highly effective for standard benchmarks like CIFAR-10 and MNIST, they show some deviations in more complex settings like ImageNet with ResNet-50, likely due to these models being less overparameterized. Future research will focus on integrating $AutoL_2$ with standard learning rate schedules and exploring finite-width corrections to the infinite-width theory.
Summary updated