How to set AdamW's weight decay as you scale model and dataset size

Xi Wang, Laurence Aitchison

Abstract

Weight decay is a critical hyperparameter for training deep neural networks, yet its scaling behavior remains poorly understood. This paper demonstrates that AdamW's weight decay can be re-parameterized as an Exponential Moving Average (EMA) timescale. By maintaining a constant timescale in terms of epochs ($\tau_{epoch}$), we derive simple scaling rules: weight decay should be inversely proportional to dataset size and directly proportional to model width when using Maximal Update Parameterization ($\mu$P). Empirical results across ResNet, ViT, and GPT architectures confirm that these rules stabilize the optimal hyperparameter landscape during scaling.

Introduction

In the era of "Scaling Laws," researchers have become experts at transferring hyperparameters from small prototype models to massive production runs. Frameworks like Maximal Update Parameterization ($\mu$P) have provided robust methods to scale learning rates. However, weight decay—the crucial regularization term—has remained a "black box," often requiring expensive grid searches for every new model or dataset size. This work fills that gap by providing a theoretical and empirical recipe for scaling weight decay in the AdamW optimizer.

The EMA Perspective on Weight Decay

The core innovation of this work is a change in perspective: in AdamW, the weights themselves behave like an Exponential Moving Average (EMA) of the gradient updates. The key hyperparameter for any EMA is its timescale ($\tau$), which determines how many past iterations the average "remembers." In AdamW, this timescale is defined as:

$$\tau_{iter} = \frac{1}{\eta \lambda}$$

where $\eta$ is the learning rate and $\lambda$ is the weight decay coefficient. The authors hypothesize that the optimal timescale measured in epochs ($\tau_{epoch}$) is a fundamental property of the task and should remain constant even as the scale changes. This means the model should always "remember" the same fraction of the dataset regardless of its size.

Derived Scaling Rules

From the requirement of a constant $\tau_{epoch}$, two primary scaling rules emerge:

  1. Dataset Scaling: If the dataset size ($N$) increases while the learning rate is fixed, weight decay should decrease proportionally: $\lambda \propto 1/N$.
  2. Model Width Scaling: When using $\mu$P, which suggests decreasing the learning rate as width increases ($\eta \propto 1/width$), weight decay must increase linearly with width ($\lambda \propto width$) to keep the timescale stable.

Empirical Results

The authors validated these rules across diverse architectures (ResNet-18, ViT, NanoGPT) and datasets (CIFAR-10, ImageNet, OpenWebText).

Figure 2: Comparison of tuning weight decay ($\lambda$) versus tuning the EMA timescale ($\tau_{epoch}$) across different dataset sizes. When parameterizing by $\tau_{epoch}$, the optimal values align vertically, indicating stability across scales.

As shown in the figure above, when tuning $\lambda$ directly, the optimal value shifts significantly as the dataset size changes (top rows for ResNet and ViT). However, when the experiments are re-parameterized by $\tau_{epoch}$, the optima align almost perfectly across all dataset sizes. This stability holds across training and test accuracy, as well as loss metrics.

Figure 6: Detailed view of ResNet test loss alignment under the $\tau_{epoch}$ parameterization.

Restoring $\mu$P Stability

A significant finding is that standard $\mu$P learning rate transfer can break down for AdamW if weight decay is held constant. As models get wider, the optimal learning rate shifts unless weight decay is also scaled to fix the EMA timescale. Scaling $\lambda$ linearly with width restores the predictability of the optimal learning rate across model sizes, ensuring that the model's outputs stay at a consistent scale ($O(1)$).

Conclusion

By viewing weight decay through the lens of an EMA timescale, we can move beyond heuristic tuning. Maintaining a constant $\tau_{epoch}$ ensures that the model always "remembers" the same fraction of the dataset, leading to stable hyperparameter transfer across both model width and dataset size. This framework provides a missing piece in the scaling law puzzle, enabling more efficient training of large-scale models.

Summary updated

BibTeX
@inproceedings{wang2025how,
  author = {Xi Wang and Laurence Aitchison},
  title = {{How to set AdamW's weight decay as you scale model and dataset size}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=IszVnczhfz},
}
Figure