How Weight Decay and Factorization Induce Low-Rank Weights

42 concepts28 papers1 proposal

Deep neural networks routinely operate with millions or billions of parameters, giving them the capacity to express highly complex, high-dimensional functions. Yet, instead of overfitting to noise, overparameterized models consistently find solutions that generalize remarkably well to unseen data.

A converging body of research reveals that this success is driven by an implicit low-rank bias. Rather than utilizing the full dimensionality of their parameter spaces, deep networks naturally compress their weight matrices into low-dimensional subspaces during training. This structural simplification is not an accidental byproduct of optimization, but a fundamental geometric property induced by the interplay of weight decay, matrix factorization, and stochastic optimization dynamics.

The Factorization Effect: Implicit Nuclear Norm Minimization

One of the most powerful drivers of low-rank structure occurs when a network architecture implicitly or explicitly factorizes a weight matrix. For example, in the self-attention layers of modern Transformers, the interaction between tokens is governed by the product of separate Query and Key weight matrices ($W = W_Q^T W_K$).

When we apply standard $L_2$ regularization (weight decay) to these individual factors, we are minimizing the sum of their squared Frobenius norms ($\|W_Q\|_F^2 + \|W_K\|_F^2$). Mathematically, the minimum value of this sum across all possible factorizations is exactly equal to twice the nuclear norm (the sum of the singular values) of the product matrix $W$:

$$\min_{W = W_Q^T W_K} \frac{1}{2} \left( \|W_Q\|_F^2 + \|W_K\|_F^2 \right) = \|W\|_*$$

Just as an $L_1$ penalty on network activations induces sparsity in the vector space, a nuclear norm penalty acts as an $L_1$ penalty on the singular values of a matrix, driving the less important singular values to zero.

In Weight decay induces low-rank attention layers, Kobayashi et al. prove that during gradient-based optimization, the mathematical gap between the factor norms and the product's nuclear norm shrinks at an exponential rate. This ensures that the low-rank bias of weight decay actively shapes the model's representations early in the training process, rather than only at final convergence.

This factorization mechanism is a widespread phenomenon across architectures:

Weight Decay as a Geometric Sculptor

Even in layers without explicit architectural factorization, explicit $L_2$ regularization acts as a geometric sculptor that actively strips away redundant dimensions.

When a network is trained via Stochastic Gradient Descent (SGD), the weight updates are driven by the loss gradient, which is inherently restricted to the subspace spanned by the training data. In Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networks, Yi et al. explain that weight decay acts as a continuous compression force, shrinking the weights outside of this gradient-spanned subspace. Without explicit weight decay, optimization alone is insufficient to reduce the stable rank; models retain high-rank representations that memorize noise.

High-Rank Initialization ──> [ SGD Updates + Weight Decay ] ──> Low-Rank Subspace
   (Spreads across all          (Shrinks weights outside           (Weights align with
    available dimensions)        the gradient subspace)             dominant task signals)

At the stationary points of the regularized loss function, this compression forces the weight matrices to align tightly with the loss gradient. As proved by Kuzborskij et al. in Low-rank bias, weight decay, and model merging in neural networks, this alignment simplifies the network's internal representations, resulting in an effective rank that is inversely proportional to the square root of the regularization strength ($\text{Rank} \propto 1/\sqrt{\lambda}$).

This continuous rank reduction has profound effects on the internal geometry of the network:

Optimization Dynamics: Noise and Non-Smoothness

Beyond explicit regularization penalties, the optimization process itself introduces an implicit bias toward low-rank configurations.

A primary driver of this behavior is the stochasticity inherent to mini-batch updates. In SGD vs GD: Rank Deficiency in Linear Networks, Sagitova et al. contrast standard Gradient Descent (GD) with SGD. While GD largely preserves the complexity and rank of its initialization, the stochastic noise in SGD creates a repulsive force between the eigenvalues of the weight matrix. This force actively drives irrelevant dimensions toward zero, acting as a minimalist editor that enforces rank deficiency across both linear and non-linear networks.

Furthermore, the interplay between gradient descent and non-smooth activation functions (like ReLU or Leaky ReLU) accelerates this collapse when learning from structured data. As shown by Kou et al. in Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data, optimizing overparameterized non-smooth networks on nearly-orthogonal data naturally biases the weight matrices toward low-rank structures—collapsing completely to a rank-1 matrix for Leaky ReLU, and remaining bounded by a small constant for standard ReLU.

Why Low-Rank Bias Matters

Recognizing that optimization and regularization conspire to minimize rank provides a unified lens for understanding several distinct deep learning phenomena.

Grokking as a Complexity Phase Transition

The mysterious phenomenon of "grokking"—where a network suddenly transitions from perfect training memorization to perfect validation generalization long after overtraining—can be explained via rank dynamics. In The Complexity Dynamics of Grokking, DeMoss et al. track the network's intrinsic complexity using spectral entropy. They show that the onset of generalization is marked by a sharp collapse in the effective rank of the weight matrices, signaling that the model has successfully shed its high-rank, noisy memorization patterns in favor of a low-dimensional logical structure.

Permissive Model Merging

Model merging allows practitioners to combine two independent networks trained on different tasks by simply averaging or summing their weights. Kuzborskij et al. demonstrate that this is possible precisely because of the low-rank bias induced by weight decay. Because the weights of each model are compressed into narrow, task-specific subspaces, networks trained on nearly orthogonal inputs do not interfere with each other's representations when their matrices are summed.

Robustness, Pruning, and Generalization Bounds

The compression of weights into low-rank configurations fundamentally alters how models handle information loss and perturbations:

To formalize this, Ledent et al. introduce the first theoretical generalization bounds in Generalization Bounds for Rank-sparse Neural Networks that explicitly exploit this spontaneous "rank collapse" or "bottleneck rank" property. By accounting for the true, low-rank nature of the learned representations rather than the raw parameter count, these bounds provide a much tighter, mathematically rigorous explanation for why deep networks generalize so effectively.

Key Foundational Work

LocalDrop: A Hybrid Regularization for Deep Neural Networks (2022) — contributed 1 concept elements to this direction.
Neural Similarity Learning (2019) — contributed 1 concept elements to this direction.
Meta-free few-shot learning via representation learning with weight averaging (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_low_rank_bias_c48bc223,
  title = {implicit low-rank bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-low-rank-bias/rdr_c48bc223160945cbb8130b63925126dd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_implicit_low_rank_bias_c48bc223_v1,
  title = {implicit low-rank bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-low-rank-bias/rdr_c48bc223160945cbb8130b63925126dd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake