Grams: Gradient Descent with Adaptive Momentum Scaling

Yang Cao, Xiaoyu Li, Zhao Song

Abstract

We introduce Gradient Descent with Adaptive Momentum Scaling (Grams), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning. Unlike traditional optimizers that directly integrate momentum into updates, Grams separates the update direction, derived from current gradients, from momentum, which is used solely for adaptive magnitude scaling. This approach enables Grams to achieve improved loss descent compared to state-of-the-art cautious and momentum-based optimizers. We establish a global convergence guarantee for Grams and validate its effectiveness through extensive empirical evaluations. The results demonstrate Grams' superior performance, including faster convergence and better generalization, compared to widely-used optimizers such as Adam, Lion, and their cautious variants. Our results highlight Grams' potential as a transformative approach for efficient optimization in large-scale machine learning.

Introduction

Training Large Language Models (LLMs) is a computationally intensive task where even marginal gains in optimizer efficiency can lead to significant savings in time and resources. Traditional adaptive optimizers like Adam and Lion rely on "momentum"—a moving average of past gradients—to smooth updates. However, these methods couple momentum and gradients, meaning the update direction is a mixture of current and historical gradient information. This coupling can lead to "momentum overshoot," where the optimizer continues moving in an outdated direction despite changes in the local loss landscape, potentially slowing convergence or missing sharp minima. While "Cautious" optimizers (e.g., C-Adam) attempt to address this by masking momentum when it conflicts with the current gradient, Grams proposes a more fundamental decoupling.

Methodology: Decoupling Direction and Magnitude

The core innovation of Grams is the explicit separation of an update's direction from its magnitude. While standard optimizers use momentum to steer the update, Grams uses the current gradient solely to determine the direction and momentum solely to scale the step size.

Specifically, Grams maintains first ($m_t$) and second ($v_t$) moment estimates. The update vector $\hat{u}_t$ is calculated as:

$$\hat{u}_t = \text{sign}(g_t) \circ |u_t|$$

where $g_t$ is the current gradient and $u_t$ is the magnitude derived from standard momentum and second-moment estimates. By forcing the direction to match $\text{sign}(g_t)$, Grams ensures that every step aligns with the local gradient evidence, preventing the "zigzagging" or "overshooting" characteristic of traditional momentum-heavy methods.

Optimization trajectories showing Grams' direct path to the optimum compared to the overshooting or zigzagging of other methods like Lion and Adam.

Theoretical analysis using Hamiltonian dynamics demonstrates that Grams achieves a strictly better descent in the loss landscape compared to Cautious Adam under identical step-size conditions. Furthermore, the authors establish a global convergence guarantee for Grams under standard L-smoothness and PL-condition assumptions.

Experimental Evaluation

The effectiveness of Grams was evaluated across several standard benchmarks:

  • C4 (English subset): Used for pre-training a Llama 60M model from scratch.
  • CIFAR-10: Used for training a WideResNet-50-2 model to test computer vision performance.
  • MetaMathQA: Used for full fine-tuning and Parameter-Efficient Fine-Tuning (PEFT) of Llama 3.2 models (1B and 3B versions).
  • GSM-8K and MATH: Used as evaluation benchmarks for the fine-tuned models to test mathematical reasoning.

Key Results

Grams consistently outperformed its peers across all experimental setups. In LLM pre-training, Grams achieved a perplexity of 38.60, significantly lower than Adam (49.83) and Lion (50.25). On CIFAR-10, Grams reached 90.55% accuracy, surpassing Adam (87.56%) and its cautious variants.

Convergence plot demonstrating Grams' significantly faster and deeper loss descent over 400 steps compared to Adam, Lion, and their variants.

In fine-tuning tasks, Grams hit 51.02% accuracy on the GSM-8K task with the Llama 3.2 1B model, outperforming Adam (48.90%). These results underscore Grams' ability to maintain directional integrity, leading to faster convergence and improved generalization.

Conclusion and Future Work

Grams represents a transformative approach to optimization by decoupling update direction from magnitude. By ensuring every step aligns with the local gradient, it avoids the pitfalls of momentum overshoot and achieves superior performance in both language and vision tasks. While the early-stage advantage is clear, future research will focus on evaluating Grams' stability in trillion-token pre-training runs and exploring its synergy with other optimization techniques such as ADOPT or SOAP-Muon.

Summary updated

Abstract

We introduce \textbf{Gr}adient Descent with \textbf{A}daptive \textbf{M}omentum \textbf{S}caling (\textbf{Grams}), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning. Unlike traditional optimizers that directly integrate momentum into updates, Grams separates the update direction, derived from current gradients, from momentum, which is used solely for adaptive magnitude scaling. This approach enables Grams to achieve improved loss descent compared to state-of-the-art cautious and momentum-based optimizers. We establish a global convergence guarantee for Grams and validate its effectiveness through extensive empirical evaluations. The results demonstrate Grams' superior performance, including faster convergence and better generalization, compared to widely-used optimizers such as Adam, Lion, and their cautious variants. Our results highlight Grams' potential as a transformative approach for efficient optimization in large-scale machine learning.
BibTeX
@inproceedings{cao2025grams,
  author = {Yang Cao and Xiaoyu Li and Zhao Song},
  title = {{Grams: Gradient Descent with Adaptive Momentum Scaling}},
  booktitle = {SCOPE @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=GmKQnpQdsc},
}
Figure