Square Root Learning Rate Scaling for Adaptive Optimizers

19 concepts11 papers1 proposal

When scaling up deep learning training across parallel hardware, the most direct lever practitioners have is increasing the mini-batch size. However, simply increasing the batch size without adjusting other hyperparameters alters the optimization dynamics, often leading to a "generalization gap" where the model fails to perform as well as one trained with smaller batches.

To maintain identical training trajectories as the batch size $B$ scales, we must adjust the learning rate $\eta$. For standard Stochastic Gradient Descent (SGD), a well-established heuristic dictates a linear scaling rule: if you multiply the batch size by a factor of $k$, you should multiply the learning rate by $k$.

However, a growing body of research demonstrates that for adaptive optimizers like Adam, RMSprop, and LAMB, this linear rule fails. Instead, adaptive optimizers require a square-root scaling rule: when the batch size increases by a factor of $k$, the learning rate must scale by $\sqrt{k}$.

Why the Optimizer Changes the Scaling Math

To understand why adaptive optimizers scale differently than SGD, we have to look at how they process gradient variance.

In standard SGD, the parameter update is directly proportional to the mini-batch gradient. When we increase the batch size, we average more samples, which reduces the variance (or noise) of our gradient estimate linearly. To compensate for this drop in noise and keep the optimization trajectory stochastic rather than purely deterministic, the step size must increase linearly. As shown by Diego Granziol et al. using Random Matrix Theory, this linear relationship is mathematically native to SGD but fundamentally breaks under coordinate-wise normalization.

Adaptive optimizers do not update parameters using the raw gradient. Instead, they divide the gradient by the square root of its uncentered second moment (a moving average of the squared gradients, denoted as $\sqrt{v_t}$). This coordinate-wise normalization alters how batch size changes propagate through the update step.

As Jacob Hilton et al. demonstrate, achieving batch size-invariance requires accounting for how gradient noise interacts with these second-moment estimates. Because the normalization step itself involves a square root, the scaling factor required to keep the update variance stable scales with $\sqrt{k}$ rather than $k$.

The Continuous-Time Perspective

We can gain deeper intuition by looking at optimization through Stochastic Differential Equations (SDEs), which model discrete gradient steps as continuous-time trajectories.

Discrete Steps:  [θ_0] --(η·g_1)--> [θ_1] --(η·g_2)--> [θ_2]
                       \               \               \
Continuous SDE:         \_______________v_______________v_______> dθ_t = μ(θ_t)dt + σ(θ_t)dW_t

By analyzing these continuous-time approximations, Sadhika Malladi et al. proved that the continuous mapping of discrete steps in adaptive algorithms is proportional to the square of the learning rate ($\eta^2$), rather than a linear dependence on $\eta$. To keep the continuous trajectory invariant when changing the batch size, the discrete step size must adjust according to a square root rule.

Similarly, Enea Monzio Compagnoni et al. studied the role of mini-batch noise using SDEs. They found a linear relationship between asymptotic loss and noise levels in adaptive methods, which contrasts sharply with the quadratic scaling observed in SGD. This structural difference implies that to maintain optimization stability and preserve the trajectory, both the learning rate and weight decay must be scaled by the square root of the batch size change factor.

Preserving the Noise-to-Signal Ratio

Why does keeping this trajectory invariant matter so much for final model performance? The answer lies in the geometry of the loss landscape.

Elad Hoffer et al. argue that the generalization gap in large-batch training is not an inherent flaw of large batches, but rather a consequence of a mismatch in training dynamics. When batch sizes increase without proper hyperparameter scaling, the optimization process becomes too deterministic, causing the model to miss flatter, more generalizable minima. Adjusting the learning rate by the square root of the batch size maintains a constant noise-to-signal ratio in the weight updates, ensuring that the stochasticity of the optimization process remains consistent.

This balancing act has enabled massive scaling breakthroughs across various domains:

  • Language Modeling: Yang You et al. utilized a square-root scaling heuristic within the LAMB optimizer to scale BERT training to massive batch sizes without losing accuracy, reducing training time from three days to 76 minutes. More recently, William Merrill et al. showed that square-root scaling allows Adam to maintain optimization stability and recover loss parity up to a model's Critical Batch Size.
  • Reinforcement Learning: In offline reinforcement learning, where scaling batch sizes can drastically reduce training times, Alexander Nikulin et al. applied a square-root learning rate adjustment to scale mini-batch sizes up to 10,000 stably, bypassing the need for complex warm-up schedules.
  • Physics-Informed Neural Networks (PINNs): Shyam Sankaran et al. validated that comparing linear and square-root scaling schemes is essential for scaling PINN training, where larger batch sizes consistently improve both convergence speed and final accuracy.

Nuances and Open Challenges

While the square-root scaling rule provides a robust baseline for adaptive optimizers, recent work suggests the relationship can be more complex in practice.

Shuaipeng Li et al. uncovered a "surge phenomenon" in Adam-style optimizers. Unlike SGD, where the optimal learning rate scales monotonically with batch size, Adam's optimal learning rate can exhibit a non-monotonic curve—first increasing and then decreasing as the batch size expands. This indicates that a simple square-root rule may not capture the entire optimization landscape at extreme scales.

Furthermore, the necessity of the square root itself is being re-examined. Wu Lin et al. suggest that the square-root formulation in adaptive methods was originally a heuristic designed to stabilize training under fixed learning rates. In modern training regimes that leverage advanced schedules and learning rate decay, they argue this square-root dependency might introduce unnecessary side effects, opening up new avenues for designing batch-size invariant optimizers from a second-order perspective.

Key Foundational Work

Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017) — contributed 2 concept elements to this direction.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_square_root_learning_rate_scaling_ee3ab9dd,
  title = {square-root learning rate scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/square-root-learning-rate-scaling/rdr_ee3ab9dde1c24ab98497b2b358f7e1fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_square_root_learning_rate_scaling_ee3ab9dd_v1,
  title = {square-root learning rate scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/square-root-learning-rate-scaling/rdr_ee3ab9dde1c24ab98497b2b358f7e1fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake