Weight decay induces low-rank attention layers

Seijin Kobayashi, Yassir Akram, Johannes von Oswald

Introduction

Weight decay, or $L2$ regularization, is a ubiquitous technique in neural network training designed to improve generalization by penalizing large parameter values. While its effects on linear layers are well-understood, its impact on the attention mechanism—the core component of modern Transformer architectures—remains less clear. This research demonstrates that weight decay on the factorized matrices of attention layers (e.g., $W_K, W_Q, W_V, P$) acts as an implicit nuclear norm regularizer, inducing a low-rank bias that can significantly constrain model capacity.

Theoretical Framework: The Hidden Nuclear Norm

The fundamental insight of this work is that when weight decay is applied to two matrices $A$ and $B$ whose product $AB^T$ is used in the model, the regularization effect is not merely on the individual weights. Mathematically, the authors prove that the local minima of an $L2$-regularized factorized system are identical to the minima of a system regularized by the nuclear norm of their product: $$\min_{A,B} \mathcal{L}(AB^T) + \frac{\lambda}{2}(\|A\|_F^2 + \|B\|_F^2) \iff \min_{X} \mathcal{L}(X) + \lambda \|X\|_*$$ where $\|X\|_*$ is the nuclear norm (the sum of singular values). Since the nuclear norm is a convex proxy for matrix rank, weight decay effectively "squeezes" the rank of attention layers.

The discrepancy between the sum of squared Frobenius norms and the nuclear norm of the product vanishes exponentially fast during training.

Dynamics of Regularization

Beyond static minima, the researchers analyzed the gradient flow of these systems. They found that the discrepancy between the $L2$ penalty on factors and the nuclear norm of the product vanishes exponentially fast during optimization. This implies that the low-rank pressure is active from the earliest stages of training, rather than being a late-stage convergence phenomenon. This rapid "balancing" of factor matrices ($A^T A \approx B^T B$) was observed across various scales, from synthetic tasks to large language models.

Empirical Evidence and Rank Reduction

The theoretical predictions were validated through extensive experiments. By training Transformers on tasks like associative recall and language modeling (The Pile), the authors observed a direct correlation between the weight decay coefficient $\lambda$ and the resulting rank of the attention matrices.

Singular values $s_i$ of the product matrix decrease as the weight decay coefficient $\lambda$ increases, eventually leading to rank collapse.

Analysis of pre-trained foundation models, such as LLaMA 2 and Vision Transformers (ViT), further confirmed these findings. In these models, the Frobenius norms and nuclear norms of attention layers almost perfectly coincide, suggesting that the massive scale of their training has already driven them into this low-rank, balanced regime.

Implications for Transformer Training

The discovery that weight decay acts as a rank-reduction tool has significant practical implications. While low-rank structures can be efficient, they may also act as a bottleneck for the expressive power of attention layers. The authors found that while high weight decay is beneficial for Feed-Forward (MLP) layers, it can be detrimental to attention layers.

Their results suggest that a decoupled regularization strategy—applying lower weight decay to attention parameters than to MLP parameters—can lead to better performance, such as lower perplexity in language models. This highlights the need for layer-specific optimization strategies to maintain the necessary representational complexity in modern AI architectures.

Summary updated

Abstract

The effect of regularizers such as weight decay when training deep neural networks is not well understood. We study the influence of weight decay as well as $L2$-regularization when training neural network models in which parameter matrices interact multiplicatively. This combination is of particular interest as this parametrization is common in attention layers, the workhorse of transformers. Here, key-query, as well as value-projection parameter matrices, are multiplied directly with each other: $W_K^TW_Q$ and $PW_V$. We extend previous results and show on one hand that any local minimum of a $L2$-regularized loss of the form $L(AB^\top) + \lambda (\|A\|^2 + \|B\|^2)$ coincides with a minimum of the nuclear norm-regularized loss $L(AB^\top) + \lambda\|AB^\top\|_*$, and on the other hand that the 2 losses become identical exponentially quickly during training. We thus complement existing works linking $L2$-regularization with low-rank regularization, and in particular, explain why such regularization on the matrix product affects early stages of training. Based on these theoretical insights, we verify empirically that the key-query and value-projection matrix products $W_K^TW_Q, PW_V$ within attention layers, when optimized with weight decay, as usually done in vision tasks and language modelling, indeed induce a significant reduction in the rank of $W_K^TW_Q$ and $PW_V$, even in fully online training. We find that, in accordance with existing work, inducing low rank in attention matrix products can damage language model performance, and observe advantages when decoupling weight decay in attention layers from the rest of the parameters.
BibTeX
@inproceedings{kobayashi2024weight,
  author = {Seijin Kobayashi and Yassir Akram and Johannes von Oswald},
  title = {{Weight decay induces low-rank attention layers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=oDeqjIM9Sk},
}
Figure