Deep neural networks often contain millions more parameters than they have training examples, yet they generalize remarkably well. A growing body of research suggests this success isn't a mystery of "overparameterization" alone, but rather the result of an implicit low-rank bias. This bias forces high-dimensional weight matrices to collapse into simpler, low-dimensional structures during training, effectively reducing the functional complexity of the model.
While we often view weight decay ($L_2$ regularization) as a simple tool to prevent weights from exploding, it actually acts as a geometric sculptor, carving out low-rank representations that are essential for generalization, model merging, and efficiency.
Weight Decay as a Rank Minimizer
The traditional view of weight decay is that it penalizes the magnitude of weights to prevent overfitting. However, Chugang Yi, Haizhao Yang, and Ke Chen demonstrate that weight decay does something much more specific: it induces a low-rank bias in weight matrices.
By shrinking weights toward the subspace spanned by recent gradient updates, weight decay effectively compresses the model's internal representations. Tomer Galanti and colleagues argue that SGD and weight decay "secretly" minimize the rank of the network. This mechanism explains why models can be pruned or compressed after training without losing accuracy—the training process itself has already made them low-rank.
The Factorization Trick: From $L_2$ to Nuclear Norm
One of the most powerful insights in this field is the mathematical equivalence between standard regularization on factorized layers and more complex rank-minimization techniques.
When we represent a weight matrix $W$ as a product of two smaller matrices $U$ and $V$ (as in LoRA or factorized layers), applying standard $L_2$ weight decay to $U$ and $V$ individually is mathematically equivalent to minimizing the nuclear norm (the sum of singular values) of the product $W = UV^T$. Uijeong Jang, Jason D. Lee, and Ernest K. Ryu show that this nuclear norm minimization is what allows LoRA to find high-quality solutions despite its non-convex landscape.
This same principle applies to Transformers. Heejune Sheen and Siyu Chen prove that training the Query and Key matrices separately implicitly minimizes the nuclear norm of their product. This explains why attention layers naturally tend toward low-rank structures, acting as "token selectors" that filter out noise.
Depth as a Regularizer
Intuitively, adding more layers to a network should make it more complex. However, if those layers are linear, they don't change what the network can represent, but they fundamentally change what it prefers to learn.
Greg Ongie and Rebecca Willett have shown that increasing the number of linear layers introduces a stronger inductive bias toward low-rank representations. Mathematically, weight decay in a deep linear stack corresponds to minimizing a Schatten quasi-norm of the end-to-end product matrix. As the network gets deeper, this penalty becomes more aggressive, pushing the model toward even lower-rank solutions.
This "depth-induced" regularization is the core idea behind the Implicit Rank-Minimizing Autoencoder (IRMAE) by Li Jing, where simply adding redundant linear layers forces the autoencoder to learn a compact, low-dimensional latent space without any explicit bottleneck.
The Tension with Neural Collapse
This low-rank bias also interacts with Neural Collapse, a phenomenon where features of the same class cluster into a single point (a "simplex") in the final layer. Emanuele Zangrando and colleagues prove that weight decay acts as the glue connecting feature clustering to weight simplification.
However, there is a subtle tension here. While the final layer might collapse into a high-rank symmetric structure, Peter Súkeník and Marco Mondelli argue that in deep networks, multi-layer weight decay actually favors lower-rank structures over the "perfect" symmetry of neural collapse. This suggests that the internal layers of a network are even simpler than the final output suggests, prioritizing efficiency and noise-filtering over geometric perfection.
Why It Matters: Merging and Plasticity
Understanding this low-rank bias isn't just theoretical; it explains why modern deep learning techniques work:
- Model Merging: Ilja Kuzborskij and Yasin Abbasi-Yadkori show that because weight decay forces models into narrow, low-dimensional subspaces, we can merge two different models by simply averaging their weights if their tasks are sufficiently distinct.
- Plasticity: Tessa Han and Sham M. Kakade found that high weight decay improves model plasticity. By pushing attention matrices toward low-rank structures during pretraining, the model avoids "memorizing" noise and keeps its capacity open for learning new tasks later.
- Linear Mode Connectivity: The low-rank structure induced by weight decay is essential for Linear Mode Connectivity, allowing independently trained wide networks to inhabit a shared loss basin.
In summary, the "magic" of deep learning's generalization often boils down to a simple mathematical pressure: weight decay and depth work together to ensure that even the largest models learn the simplest possible representations.