The Equivalence of Momentum and Learning Rate Scaling

21 concepts13 papers1 proposal

In deterministic optimization, momentum is celebrated for its ability to accelerate convergence, turning a slow crawl into an accelerated sprint. However, when we transition to stochastic gradient descent (SGD), the physical intuition of a heavy ball rolling down a hill changes. A growing body of research demonstrates that in many stochastic settings, momentum does not fundamentally alter the optimization path. Instead, it acts primarily as a scalar multiplier on the step size, creating an effective learning rate.

The Mechanism of the Effective Learning Rate

To understand why momentum and learning rate are interchangeable, consider how momentum updates weights. Instead of moving directly along the current stochastic gradient with a learning rate $\eta$, momentum tracks a moving average of past gradients with a decay parameter $\beta \in [0, 1)$.

When the optimization trajectory enters a steady direction or a local valley, these accumulated updates compound. Mathematically, this creates a geometric series where the total distance traveled per step is amplified. As single-author work by Li details, this interaction collapses the two hyperparameters into a single composite metric known as the effective learning rate:

$$\eta_{\text{eff}} = \frac{\eta}{1 - \beta}$$

This formula reveals that increasing momentum (making $\beta$ closer to 1) is mathematically equivalent to boosting the base learning rate $\eta$. Yuan et al. demonstrated that in the constant step-size regime of online learning, the update trajectories of SGD with momentum and standard SGD become nearly identical when mapped through this rescaling. Similarly, in the context of decaying learning rates, Leen et al. showed that momentum serves as a scaling factor where increasing momentum can directly compensate for a sub-optimal initial learning rate. The perceived speedup from momentum is often not a unique algorithmic benefit, but simply the result of operating at a larger effective step size.

Trajectory Equivalence Across Regimes

This equivalence is not just a loose approximation; it holds rigorously across several training conditions:

  • Small Learning Rates: In regimes with small learning rates, Wang et al. showed that SGD with momentum behaves almost identically to vanilla SGD when normalized by the dampening factor. Furthermore, Lyu et al. found that the implicit bias—the specific solutions a model prefers in overparameterized landscapes—becomes indistinguishable between standard gradient descent and momentum methods under very small step sizes.
  • High-Dimensional Dynamics: When training complex models in high dimensions, Jagannath et al. proved that the trajectory of SGD with momentum can be mapped directly to standard SGD through a precise rescaling of time and step-size parameters. Momentum effectively acts as a temporal reparameterization rather than changing the geometric route.
  • Predicting Convergence: Because the interaction between $\eta$ and $\beta$ reduces to this single effective rate, Zancato et al. demonstrated that different hyperparameter configurations sharing the same effective learning rate yield nearly identical training time requirements.

When the Equivalence Shifts

While the effective learning rate rule explains much of momentum's behavior, the equivalence can shift or require more nuanced scaling depending on the training phase and landscape geometry.

Leclerc et al. observed that deep network training consists of two distinct phases. In the initial high-learning-rate phase, momentum functions purely as a step-size multiplier and can be completely substituted by an increased learning rate. However, in the later, low-learning-rate phase, momentum's role shifts to helping the optimizer navigate complex local geometry, making it harder to replace entirely.

Furthermore, the exact scaling law can depend on the model architecture and optimization objective:

  • Power-Law Scaling: Cowsik et al. proposed an optimal scaling rule where the momentum term $(1-\beta)$ scales with $\eta^{2/3}$ to balance how quickly weights settle into low-loss valleys versus how fast they move along them.
  • Intrinsic Ratios: In diagonal linear networks, Papazov et al. showed that the optimization path is governed by a single intrinsic parameter defined by the ratio of the step size to the square of the effective momentum coefficient.
  • Quadratic Minimization: When optimizing quadratic problems, Goujaud et al. found that the optimal step size under momentum acceleration is exactly twice the value of the standard Polyak step size used in non-accelerated methods.

De-biasing Optimizer Comparisons

Understanding this equivalence is crucial for fair empirical evaluations. If standard SGD and momentum methods are compared using the same nominal learning rate, standard SGD often utilizes a larger effective step size per iteration than certain momentum variants, leading to misleading conclusions about stability and speed (Yan et al.).

To isolate the true algorithmic effects of momentum from simple changes in update scale, Wang et al. introduced effective learning rate normalization, which scales the base learning rate by the momentum coefficient and batch size to ensure consistent update magnitudes. When properly rescaled to match the variance or "noise floor" of momentum methods, Ganesh et al. proved that standard SGD can replicate the exact convergence rates of accelerated momentum methods, demonstrating that hyperparameter tuning can often substitute for algorithmic acceleration.

Key Foundational Work

Optimal Stochastic Search and Adaptive Momentum (1993) — contributed 2 concept elements to this direction.
On the Influence of Momentum Acceleration on Online Learning (2016) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 13 papers
  2. Version 1 · 2026-10-04 · 14 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_momentum_learning_rate_equivalence_019b1fff,
  title = {momentum-learning rate equivalence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-learning-rate-equivalence/rdr_019b1fff9ff54706b4e733d90a12bb5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_momentum_learning_rate_equivalence_019b1fff_v2,
  title = {momentum-learning rate equivalence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-learning-rate-equivalence/rdr_019b1fff9ff54706b4e733d90a12bb5f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake