Abstract
This research investigates the mechanism by which $L_2$ regularization, or weight decay, induces a "low-rank bias" in neural networks. By constraining weight matrices toward lower-dimensional structures, weight decay provides a theoretical foundation for the success of model merging—the process of combining specialized networks through simple weight summation. The authors provide mathematical proofs for norm preservation and low-rank structure at stationary points, validated by empirical analysis of large-scale models including Phi-2 and GPT-J.
Introduction: The Mystery of Model Merging
In contemporary deep learning, two phenomena have remained largely disconnected: the implicit bias of optimization toward generalizable solutions and the empirical success of model merging. Model merging allows researchers to take two models trained on different tasks and simply sum or average their weights to create a multitask model without additional training. This paper bridges these concepts by demonstrating that weight decay fundamentally alters the internal geometry of neural networks, creating the sparse parameter landscapes necessary for additive combination.
Theoretical Framework: Weight Decay as a Rank Reducer
The central thesis of this work is that weight decay does more than just prevent overfitting by penalizing large weights; it actively pushes weight matrices toward low-rank configurations. Through stationary point analysis—studying the network state where the gradient of the regularized loss is zero—the authors prove that in deep linear networks, the weights are driven to be rank-1.
To quantify this effect in more complex architectures, the authors introduce "pseudo-rank," a metric defined as the weighted harmonic mean of stable ranks across layers. They demonstrate that pseudo-rank is bounded by a factor of $\sqrt{1/\lambda}$, where $\lambda$ represents the regularization parameter. This implies that as the strength of weight decay increases, the network's internal mathematical representation becomes progressively simpler and more structured.
Empirical Evidence in Large Language Models
The theoretical predictions are rigorously tested against state-of-the-art architectures. Analysis of models such as GPT-J and Phi-2 reveals that their weight matrices possess a "stable rank" significantly lower than their nominal dimensions. For instance, in GPT-J, which has a hidden dimension of 4096, the stable rank typically falls between 10 and 25 across most layers.
Furthermore, the authors observe "norm preservation," where the Frobenius norms of weight matrices remain remarkably consistent across different layers. This stability is a direct consequence of the alignment between parameters and the gradient of the loss at stationary points, a property predicted by their theoretical derivations.
Subspace Residency and the Mechanics of Merging
The low-rank bias leads to the concept of "Subspace Residency." Because weight decay constrains weights to a narrow, low-dimensional subspace, a model trained on Task A occupies only a small fraction of the total parameter space. If Task B involves inputs that are nearly orthogonal to those of Task A, its resulting model will reside in a different, non-overlapping subspace.
Summing these weights is effective because the low-rank nature ensures that the "signals" from different tasks rarely collide. The authors provide bounds showing that the error of a merged model is controlled by the regularization parameter and the degree of input orthogonality. Without weight decay, the residual "noise" from the network's initialization would persist, creating interference that prevents clean weight summation.
Conclusion and Future Directions
While this paper provides a robust explanation for the mechanics of model merging, it highlights the importance of input orthogonality. While this can be engineered through techniques like zero-padding, its natural prevalence in real-world multitask settings remains a subject for further investigation. Additionally, while the current theory focuses on training loss at stationary points, extending these findings to fully characterize the generalization gap is a promising avenue for future research.
Summary updated