Introduction
Weight decay, or $L2$ regularization, is a ubiquitous technique in neural network training designed to improve generalization by penalizing large parameter values. While its effects on linear layers are well-understood, its impact on the attention mechanism—the core component of modern Transformer architectures—remains less clear. This research demonstrates that weight decay on the factorized matrices of attention layers (e.g., $W_K, W_Q, W_V, P$) acts as an implicit nuclear norm regularizer, inducing a low-rank bias that can significantly constrain model capacity.
Theoretical Framework: The Hidden Nuclear Norm
The fundamental insight of this work is that when weight decay is applied to two matrices $A$ and $B$ whose product $AB^T$ is used in the model, the regularization effect is not merely on the individual weights. Mathematically, the authors prove that the local minima of an $L2$-regularized factorized system are identical to the minima of a system regularized by the nuclear norm of their product: $$\min_{A,B} \mathcal{L}(AB^T) + \frac{\lambda}{2}(\|A\|_F^2 + \|B\|_F^2) \iff \min_{X} \mathcal{L}(X) + \lambda \|X\|_*$$ where $\|X\|_*$ is the nuclear norm (the sum of singular values). Since the nuclear norm is a convex proxy for matrix rank, weight decay effectively "squeezes" the rank of attention layers.
Dynamics of Regularization
Beyond static minima, the researchers analyzed the gradient flow of these systems. They found that the discrepancy between the $L2$ penalty on factors and the nuclear norm of the product vanishes exponentially fast during optimization. This implies that the low-rank pressure is active from the earliest stages of training, rather than being a late-stage convergence phenomenon. This rapid "balancing" of factor matrices ($A^T A \approx B^T B$) was observed across various scales, from synthetic tasks to large language models.
Empirical Evidence and Rank Reduction
The theoretical predictions were validated through extensive experiments. By training Transformers on tasks like associative recall and language modeling (The Pile), the authors observed a direct correlation between the weight decay coefficient $\lambda$ and the resulting rank of the attention matrices.
Analysis of pre-trained foundation models, such as LLaMA 2 and Vision Transformers (ViT), further confirmed these findings. In these models, the Frobenius norms and nuclear norms of attention layers almost perfectly coincide, suggesting that the massive scale of their training has already driven them into this low-rank, balanced regime.
Implications for Transformer Training
The discovery that weight decay acts as a rank-reduction tool has significant practical implications. While low-rank structures can be efficient, they may also act as a bottleneck for the expressive power of attention layers. The authors found that while high weight decay is beneficial for Feed-Forward (MLP) layers, it can be detrimental to attention layers.
Their results suggest that a decoupled regularization strategy—applying lower weight decay to attention parameters than to MLP parameters—can lead to better performance, such as lower perplexity in language models. This highlights the need for layer-specific optimization strategies to maintain the necessary representational complexity in modern AI architectures.
Summary updated