Low-Rank Bias, Weight Decay, and Model Merging in Neural Networks

Ilja Kuzborskij, Yasin Abbasi-Yadkori

TL;DR

Weight decay ($L_2$ regularization) does more than just prevent overfitting—it actively biases neural networks toward learning low-rank weight matrices. This paper demonstrates that this low-rank bias explains the surprising success of model merging, showing that networks trained on different, nearly orthogonal tasks can be successfully combined by simply summing their weight matrices together.

What Problem Does This Paper Address?

Model merging—the practice of combining multiple trained neural networks by averaging or adding their weights—has become a popular and computationally cheap way to create multitask models. Despite its empirical success, we still lack a complete theoretical understanding of why it works. Why don't the parameters of two independently trained networks interfere with each other and destroy the model's performance when combined?

Previous deep learning theory often studied "implicit bias" under idealized settings without explicit regularization, or assumed the models achieved perfect zero training error (interpolation). This paper addresses this gap by analyzing the explicit role of $L_2$ regularization (weight decay) at stationary points. The authors connect weight decay to a low-rank structural bias and use it to provide a mathematical foundation for model merging.

Methods

The authors mathematically analyze depth-$K$ neural networks with $H$-positive homogeneous activations (such as ReLU or powers of ReLU) trained using logistic loss and $L_2$ regularization. They focus on stationary points—the limiting solutions where the gradient of the regularized loss function vanishes.

Key methodological contributions include:

  1. Characterizing Low-Rank Structure: The authors introduce a bound on the "pseudo-rank" (a weighted harmonic mean of the stable ranks across layers) and show it is explicitly controlled by the regularization parameter ($\lambda$) and the training loss.
  2. Proving Norm Preservation: They prove that at stationary points, the Frobenius norms of weight matrices are preserved across layers up to a scaling factor determined by the network's architecture.
  3. Model Merging Framework: They analyze a multitask setting where two networks are trained on distinct datasets with approximately orthogonal inputs. They provide bounds proving that the sum of these two networks' weights performs well on both tasks.

Datasets

The paper validates its theoretical claims using a mix of synthetic benchmarks, classic datasets, and large-scale pre-trained models:

  • Synthetic Data: Inputs generated from isotropic Gaussians to simulate perfectly orthogonal tasks.
  • Fashion MNIST: Adapted for binary classification and modified by appending/prepending zeros to ensure input orthogonality between different tasks.
  • Pre-trained Transformers: To see if these theoretical properties hold in the wild, the authors evaluate several publicly available models: BERT (~110M parameters), RoBERTa (~125M), GPT2-Large (~774M), GPT2-XL (~1.5B), Phi-2 (~2.8B), and GPT-J (~6B).

Key Insights and Ideas

  • Weight Decay is a Rank Compressor: The primary insight is that weight decay forces weight matrices to have a low stable rank, meaning their variance is concentrated across only a few dominant eigenvectors.
  • Orthogonal Tasks Inhabit Non-Overlapping Subspaces: Because of this low-rank bias, when two networks are trained on nearly orthogonal tasks, their weight matrices learn to "reside" in entirely different, non-interfering subspaces of the parameter space.
  • Summation vs. Averaging: While standard model merging often relies on parameter averaging for models trained on similar tasks, this paper reveals that for distinct, orthogonal tasks, parameter summation is the correct mathematical operation to preserve performance.

Key Results

  • Low Rank in Large Language Models: Even though modern transformers violate several theoretical assumptions (e.g., they use LayerNorm and residual connections), empirical evaluations show they still exhibit remarkably low stable ranks across almost all layers.
Stable ranks and Frobenius norms of different weight matrices in pretrained Phi-2 and GPT-J models.
  • L2 Penalty Drives Down Rank: Experiments on synthetic data confirm that as the $L_2$ penalty increases, the average inverse stable rank increases (meaning the actual rank decreases), validating the core theoretical bound.
Low rank induced by weight decay as L2 penalty increases.
  • Successful Merging: On both synthetic data and the modified Fashion MNIST dataset, summing the weights of networks trained on orthogonal tasks resulted in a merged model that achieved near-zero training loss and preserved generalization on both tasks simultaneously. Conversely, merging models trained on the same task via summation failed completely.

Limitations and Future Work

While the empirical results on LLMs are promising, the strict mathematical proofs rely on the assumption of positive homogeneity (e.g., pure ReLU networks without residual connections or normalization layers). Modern architectures break these assumptions. Additionally, the merging theory strictly requires the input spaces of the different tasks to be approximately orthogonal. Future work is needed to extend these proofs to non-orthogonal task distributions and more complex, heterogeneous network architectures.

Summary updated

Abstract

We explore the low-rank structure of the weight matrices in neural networks at the stationary points (limiting solutions of optimization algorithms) with $L2$ regularization (also known as weight decay). We show several properties of such deep neural networks, induced by $L2$ regularization. In particular, for a stationary point we show alignment of the parameters and the gradient, norm preservation across layers, and low-rank bias: properties previously known in the context of solutions of gradient descent/flow type algorithms. Experiments show that the assumptions made in the analysis only mildly affect the observations. In addition, we investigate a multitask learning phenomenon enabled by $L2$ regularization and low-rank bias. In particular, we show that if two networks are trained, such that the inputs in the training set of one network are approximately orthogonal to the inputs in the training set of the other network, the new network obtained by simply summing the weights of the two networks will perform as well on both training sets as the respective individual networks. We demonstrate this for shallow ReLU neural networks trained by gradient descent, as well as deep linear networks trained by gradient flow.
BibTeX
@inproceedings{kuzborskij2026low,
  author = {Ilja Kuzborskij and Yasin Abbasi-Yadkori},
  title = {{Low-Rank Bias, Weight Decay, and Model Merging in Neural Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=EjOV1vJfGi},
}
Figure