No article text was recorded for this version.
width-dependent learning rate scaling
Version 1 of 19 papers15 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Adaptive Optimization in the ∞-Width Limit
- Blog: How to set AdamW's weight decay as you scale model and dataset size
- Blog: Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
- Blog: Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
- Blog: Scalable GANs with Transformers
- Blog: Don't be lazy: CompleteP enables compute-efficient deep transformers
- Blog: The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
- Blog: Strength of Minibatch Noise in SGD
- Blog: Scaling Exponents Across Parameterizations and Optimizers
History & citation
Current · Permanent link
- Scaling Weight Decay for Predictable Model Growth merged into this direction (57% overlap)
- This direction split from maximal update parameterization (56% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_width_dependent_learning_rate_scaling_8132e67d,
title = {width-dependent learning rate scaling},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/width-dependent-learning-rate-scaling/rdr_8132e67d6ec546e0810c532cd62d2039},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 1, frozen text):
@misc{lacuna_width_dependent_learning_rate_scaling_8132e67d_v1,
title = {width-dependent learning rate scaling},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/width-dependent-learning-rate-scaling/rdr_8132e67d6ec546e0810c532cd62d2039/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}