Scaling Hyperparameters with Network Width

15 concepts9 papers1 proposal

When scaling up deep learning models, the ideal scenario is to increase the model's capacity without re-tuning hyperparameters from scratch. Frameworks like Maximal Update Parameterization ($\mu$P) provide a mathematical foundation for how weights should be initialized and how learning rates should scale as a network's width (the number of hidden channels) approaches infinity.

However, a model's optimization dynamics depend on a delicate web of interacting components: learning rates, gradient noise, weight decay, and the internal stability constants of adaptive optimizers. If we scale width while keeping these other hyperparameters fixed, the training dynamics drift, leading to exploding variance, collapsing gradients, or broken regularization. To achieve true scale invariance and stable training, the entire optimizer configuration must be treated as a width-dependent system.

Managing Gradient Noise and Signal Dominance

Why can't we use the same learning rate when a network gets wider? The core reason lies in how network width alters the variance of updates and the nature of minibatch noise.

In Stochastic Gradient Descent (SGD), updates are inherently noisy because gradients are computed over a small sample of data. As a network grows wider, the dimensionality of the parameter space expands, altering the fluctuations of the training loss. Liu Ziyin et al. demonstrate that to maintain training stability in wide networks, the learning rate must be rescaled inversely to the network width. This inverse scaling directly counteracts the explosion of training loss variance caused by the increased dimensionality.

This relationship between width and optimal noise is heavily dependent on the underlying parameterization scheme. As shown by Daniel S. Park et al., standard parameterizations allow for a constant learning rate-to-batch size ratio as width increases, whereas Neural Tangent Kernel (NTK) parameterization requires this ratio to increase linearly with width to maintain peak generalization performance.

Furthermore, the interaction between layers introduces specific stability thresholds. Tianyu Pang et al. show that width-dependent learning rate scaling establishes precise step-size thresholds based on the hidden dimension. If these thresholds are exceeded, initialization and residual noise overwhelm the true learning signal, causing unstable layer interactions and preventing convergence.

This insight is highly practical: in complex architectures like Generative Adversarial Transformers, Sangeek Hyun et al. successfully stabilized training across scales by adjusting the learning rate inversely to the number of channels, ensuring consistent update magnitudes and preventing divergence in wider models.

The Hidden Scale-Breaker: Optimizer Stability Constants

While adjusting the learning rate handles the primary update magnitude, adaptive optimizers like Adam introduce a subtle failure mode when scaling width: the numerical stability constant ($\epsilon$). In the Adam update rule, the gradient is divided by $\sqrt{v} + \epsilon$, where $v$ is the moving average of squared gradients.

In standard training, $\epsilon$ is treated as a tiny, fixed constant (e.g., $10^{-8}$) meant only to prevent division by zero. However, as the network width approaches infinity, the magnitude of the gradients ($v$) changes predictably based on the parameterization. If $\epsilon$ remains fixed, it ceases to be a negligible safeguard and instead begins to dominate the denominator.

Etai Littwin et al. mathematically formalize this in the infinite-width limit, proving that $\epsilon$ must be scaled in proportion to the network width. If left unadjusted, the fixed constant causes adaptive gradient updates to either completely collapse or explode as the parameter count grows.

This theoretical breakdown has severe practical consequences for hyperparameter transfer. Katie E Everett et al. demonstrate that these fixed numerical stability constants disrupt the mathematical scale invariance required to transfer hyperparameters from small models to wide ones. To circumvent this entirely, the authors propose scale-invariant alternatives like Adam-atan2, which eliminates $\epsilon$ altogether.

For deep architectures, adjusting these stability parameters alongside other dimensions is crucial. Nolan Simran Dey et al. show that scaling both weight decay and numerical stability parameters relative to model dimensions is essential to prevent numerical instability and ensure "non-lazy" feature learning across varying widths and depths.

Aligning the Timescale of Weight Decay

The final piece of the width-scaling puzzle is weight decay. In modern optimizers like AdamW, weight decay is decoupled from the gradient update. However, its interaction with width-scaled learning rates is often misunderstood.

To understand why weight decay must scale with width, we have to look at what weight decay actually does over time. Xi Wang et al. reframe weight decay not merely as a static regularization penalty, but as a "memory" parameter that dictates the exponential moving average (EMA) timescale of the optimizer's updates. Under standard $\mu$P, the learning rate is systematically reduced as model width increases. If you keep the weight decay coefficient constant while lowering the learning rate, the effective timescale of the optimizer's memory drifts. To keep this optimization timescale perfectly constant and prevent performance degradation, the authors show that weight decay must be scaled proportionally to the width factor.

The exact scaling behavior of weight decay depends on how the optimizer and regularization are formulated. For instance, when moving beyond AdamW to advanced matrix-preconditioned optimizers (like Shampoo or SOAP), Shikai Qiu et al. find that scaling independent weight decay inversely with model width preserves the optimal regularization balance. This simple, predictable rule ensures that the relative contribution of weight decay remains balanced against the gradient updates across scales, enabling seamless zero-shot hyperparameter transfer.

Key Foundational Work

Strength of Minibatch Noise in SGD (2022) — contributed 1 concept elements to this direction.
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_width_dependent_learning_rate_scaling_8132e67d,
  title = {width-dependent learning rate scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/width-dependent-learning-rate-scaling/rdr_8132e67d6ec546e0810c532cd62d2039},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_width_dependent_learning_rate_scaling_8132e67d_v1,
  title = {width-dependent learning rate scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/width-dependent-learning-rate-scaling/rdr_8132e67d6ec546e0810c532cd62d2039/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake