No article text was recorded for this version.
implicit low-rank bias
Version 1 of 128 papers42 conceptsfrozen 2026-10-04
Papers in this version
- Blog: SGD with Weight Decay Secretly Minimizes the Ranks of Your Neural Networks
- Blog: Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networks
- Blog: Low-rank bias, weight decay, and model merging in neural networks
- Blog: Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and Generalization Bounds
- Blog: Weight decay induces low-rank attention layers
- Blog: Grokking, Rank Minimization and Generalization in Deep Learning
- Blog: LoRA Training in the NTK Regime has No Spurious Local Minima
- Blog: SGD vs GD: Rank Deficiency in Linear Networks
- Blog: LocalDrop: A Hybrid Regularization for Deep Neural Networks
- Blog: Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
- Blog: Weight Decay Improves Language Model Plasticity
- Blog: Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
- Blog: On the Local Complexity of Linear Regions in Deep ReLU Networks
- Blog: Transformers learn through gradual rank increase
- Blog: Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data
- Blog: LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail)
- Blog: Generalization Bounds for Rank-sparse Neural Networks
- Blog: The Complexity Dynamics of Grokking
- Blog: Disentanglement via Latent Quantization
- Blog: RepSpec: Structural Re-parameterized Draft Model Training for Speculative Decoding
- Blog: Neural Similarity Learning
- Blog: Inductive Bias and Spectral Properties of Single-Head Attention in High Dimensions
- Blog: Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
- Blog: Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
- Blog: Meta-free few-shot learning via representation learning with weight averaging
- Blog: $W_K, W_V$ is Probably All You Need: On the Necessity of the Query, Key, and Value Weight Triplet in Self-Attention Transformers
- Blog: Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity
- Blog: Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks
History & citation
Current · Permanent link
- This direction split from The Low-Rank Bias of Weight Decay and Depth (61% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_implicit_low_rank_bias_c48bc223,
title = {implicit low-rank bias},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/implicit-low-rank-bias/rdr_c48bc223160945cbb8130b63925126dd},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}This exact version (version 1, frozen text):
@misc{lacuna_implicit_low_rank_bias_c48bc223_v1,
title = {implicit low-rank bias},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/implicit-low-rank-bias/rdr_c48bc223160945cbb8130b63925126dd/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}