Momentum as a Rescaled Learning Rate

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 214 papers21 conceptsfrozen 2026-10-04

In deep learning, momentum is often visualized as a "heavy ball" rolling down a loss landscape, using its mass to dampen oscillations and blast through flat plateaus. This physical intuition suggests that momentum provides a unique structural acceleration that vanilla Stochastic Gradient Descent (SGD) simply cannot replicate.

However, a growing body of research suggests this intuition may be misleading. In many practical settings, momentum does not change the path the optimizer takes; it simply changes the speed at which it travels. By understanding the Effective Learning Rate (ELR) rule, we can see that momentum is often mathematically equivalent to standard SGD with a larger step size.

The Effective Learning Rate Rule

The core idea shared across these papers is that momentum acts as a scalar multiplier for the learning rate. If you are using a learning rate $\eta$ and a momentum coefficient $\beta$ (typically 0.9), the "effective" step size you are taking is actually:

$$\eta_{eff} = \frac{\eta}{1 - \beta}$$

As Kun Yuan and Ali H. Sayed demonstrate in On the influence of momentum acceleration on online learning, momentum-based methods in online learning are functionally equivalent to standard SGD once this rescaling is applied. The perceived "speedup" from momentum is frequently just the result of an effectively increased learning rate, rather than a fundamental change in optimization dynamics.

Equivalence in Practical Regimes

This equivalence isn't just a theoretical curiosity; it holds true across several critical training scenarios:

  • Small Learning Rates: In The Marginal Value of Momentum for Small Learning Rate SGD, Runzhe Wang and Zhiyuan Li show that when learning rates are small, SGD with momentum behaves almost identically to vanilla SGD. They argue that the benefits of momentum in these regimes are marginal once the step sizes are normalized.
  • High-Dimensional Limits: Aukosh Jagannath and colleagues prove in High-dimensional limit theorems for SGD that in high-dimensional settings, momentum primarily acts as a "temporal reparameterization." The trajectory of the optimizer remains the same; momentum just changes how quickly the model moves along that trajectory.
  • Early Training Phases: Guillaume Leclerc and Aleksander Madry identify "two regimes" of training. In the initial "large-step" phase—which is crucial for generalization—momentum functions almost entirely as a scalar multiplier. In The Two Regimes of Deep Network Training, they show you can replace momentum with a larger learning rate during this phase without losing accuracy.

Why This Matters for Fair Comparisons

If momentum is just a hidden way to increase the learning rate, then many empirical comparisons between optimizers may be flawed. Bohan Wang and Wei Chen argue in When and Why Momentum Accelerates SGD that we must use "effective learning rate normalization" to fairly compare algorithms. Without this normalization, we might credit momentum for "acceleration" that could have been achieved by simply tuning the learning rate of vanilla SGD.

Similarly, Todd K. Leen and Genevieve B. Orr leveraged this relationship decades ago to create adaptive momentum algorithms. They realized that if momentum is a multiplier, it can be used to "fix" a poorly chosen initial learning rate, automatically scaling the effective step size toward the optimal $1/t$ convergence path.

When Momentum Becomes Essential

While the "rescaled SGD" perspective is powerful, it doesn't mean momentum is useless. The equivalence begins to break down in specific scenarios:

  1. Stability and Generalization: Yan Yan and Tianbao Yang point out in A Unified Analysis of Stochastic Momentum Methods that while momentum might not speed up convergence in non-convex settings, it significantly improves stability. Momentum acts as a low-pass filter on noisy gradients, which can lead to better generalization.
  2. Late-Stage Training: As Guillaume Leclerc notes, once the learning rate is reduced and the model enters the "small-step" regime to navigate fine-grained local geometry, the directional properties of momentum become more important than its role as a simple multiplier.

By viewing momentum through the lens of the Effective Learning Rate, researchers can move past the "magic ingredient" heuristic and more precisely understand how update magnitudes and directional averaging interact to drive deep learning performance.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 13 papers
  2. Version 1 · 2026-10-04 · 14 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_momentum_learning_rate_equivalence_019b1fff,
  title = {momentum-learning rate equivalence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-learning-rate-equivalence/rdr_019b1fff9ff54706b4e733d90a12bb5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_momentum_as_a_rescaled_learning_rate_019b1fff_v1,
  title = {Momentum as a Rescaled Learning Rate},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-as-a-rescaled-learning-rate/rdr_019b1fff9ff54706b4e733d90a12bb5f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}