Training massive neural networks is prohibitively expensive. To avoid wasting millions of dollars on failed runs, researchers use Maximal Update Parameterization ($\mu$P) to find optimal hyperparameters (like learning rate) on a small "proxy" model and transfer them to the full-scale version.
However, even with $\mu$P, training often becomes unstable or loses performance as models grow. Recent research reveals that the culprit is often weight decay. While often treated as a static regularization constant, weight decay must be systematically scaled alongside model width and depth to maintain the "steady state" of the network.
Weight Decay as an Optimizer Memory
To understand why weight decay needs scaling, we must first change how we view it. In the context of the AdamW optimizer, Laurence Aitchison and Xi Wang argue in How to set AdamW's weight decay as you scale model and dataset size that weight decay is best understood as an Exponential Moving Average (EMA) of recent updates.
In this view, weight decay defines a "memory timescale." It determines how long the optimizer "remembers" past gradients before they are decayed away. If you change the learning rate without adjusting weight decay, you inadvertently change this timescale. Because $\mu$P requires the learning rate to decrease as model width increases, keeping weight decay constant actually makes the optimizer's memory longer. To keep the optimization dynamics identical across scales, Aitchison and Wang show that weight decay should actually increase linearly with model width.
Maintaining Invariant Signal Scale
The primary goal of scaling rules is to ensure that the "internal physics" of the model—the magnitude of activations and the strength of gradients—remains the same regardless of size.
When we scale a model's width, the way signals propagate through layers changes. Qingyue Zhao and Quanquan Gu point out in Robust Layerwise Scaling Rules by Proper Weight Decay Tuning that standard $\mu$P can fail during the "steady state" of training. They find that for matrix parameters in architectures using LayerNorm or RMSNorm, weight decay should be scaled by the square root of the width. This specific adjustment ensures that "sublayer gains" remain invariant, preventing the signal from exploding or vanishing as it passes through the transformer blocks of a giant model.
Similarly, when "warmstarting" a large model from a smaller one, Neeratyoy Mallik and colleagues in Warmstarting for Scaling Language Models emphasize weight magnitude shrinking. If you simply copy weights from a narrow model to a wide one, the combined variance of the wider layers can cause activation norms to drift, leading to instability. Shrinking the weights acts as a manual correction to keep the model within a stable numerical range.
Scaling Beyond AdamW
The need for precise weight decay scaling extends to more advanced optimization techniques. Shikai Qiu and Andrew Gordon Wilson explored this for matrix-preconditioned optimizers like Shampoo and SOAP. In Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales, they demonstrate that scaling weight decay inversely with model width allows these complex optimizers to maintain a 1.4x speedup over AdamW even at the billion-parameter scale.
Furthermore, scaling isn't just about width. Nolan Simran Dey and the team behind Don't be lazy: CompleteP enables compute-efficient deep transformers introduced CompleteP, a framework that scales weight decay and numerical stability parameters relative to both width and depth. By ensuring that every layer—no matter how deep—contributes effectively to feature learning (avoiding "lazy" learning), they achieve significantly higher compute efficiency.
Why It Works: The Principle of Invariance
The common thread across these papers is the Principle of Invariance. Whether the scaling rule is linear, square-root, or inverse, the underlying logic is always to preserve a specific property of the training process:
- Update Invariance: The relative change in weights per step should be consistent (Aitchison & Wang).
- Activation Invariance: The scale of hidden states should not drift as layers get wider or deeper (Mallik et al., Zhao et al.).
- Noise Invariance: The ratio of "gradient signal" to "SGD noise" must be balanced to ensure generalization (Daniel S. Park et al.).
By treating weight decay as a dynamic component of the model's architecture rather than a static training trick, researchers can finally achieve the "holy grail" of deep learning: predictable, zero-shot hyperparameter transfer from small experiments to frontier-scale models.