DEAM: Accumulated Momentum with Discriminative Weight for Stochastic Optimization

Jiyang Bai, Jiawei ZhangarXiv 2019 · arXiv · OpenReview

Abstract

Optimization algorithms with momentum, such as ADAM, are cornerstones of modern deep learning due to their fast convergence. However, ADAM relies on a fixed momentum weight ($\beta_1$), which can propagate errors and cause oscillations in high-curvature regions of the loss landscape. We present Discriminative wEight on Adaptive Momentum (DEAM), an algorithm that automatically computes momentum weights based on the "discriminative angle" between updates. DEAM also introduces a backtrack term to restrict redundant updates. Our experiments show that DEAM achieves faster convergence and requires fewer hyperparameters than existing methods.

The Problem: Oscillations and Fixed Momentum

Stochastic Gradient Descent (SGD) with momentum is designed to accelerate updates in relevant directions and dampen oscillations. ADAM improves this by adding adaptive learning rates. Yet, ADAM's fixed $\beta_1$ (typically 0.9) is a "one-size-fits-all" parameter. When the optimizer encounters sharp turns or narrow valleys, this fixed weight forces the model to persist in its previous direction for too long. This leads to a "zig-zag" path, where the model overshoots the minimum and must repeatedly correct itself, wasting computational resources and slowing down convergence.

The DEAM Approach: Adaptive Momentum and Backtracking

DEAM addresses these inefficiencies through two primary innovations integrated into the ADAM framework:

1. Adaptive Momentum Weight ($\beta_{1,t}$)

Instead of a static hyperparameter, DEAM dynamically calculates the weight for the current gradient based on the discriminative angle ($\theta$) between the previous update direction and the current gradient.

  • Small Angle: If the current gradient aligns with the previous direction, momentum is stable and maintained.
  • Large Angle: If the gradient indicates a sudden change in direction, DEAM increases the weight of the current gradient to pivot the search path quickly and correct the momentum.

2. Backtrack Term ($d_t$)

When the discriminative angle exceeds $90^\circ$, it suggests a significant overshoot. DEAM employs a novel backtrack term that restricts redundant updates by effectively undoing a portion of the previous step, smoothing the optimization trajectory.

Conceptual overview of the zig-zag route and the proposed axis decomposition and backtrack mechanism.

By automating the calculation of $\beta_{1,t}$, DEAM reduces the number of hyperparameters to tune, making it more robust across different architectures. Interestingly, the authors show that if the discriminative angle is randomly distributed, the expected value of their adaptive weight is 0.1—aligning perfectly with ADAM's standard $1 - \beta_1 = 0.1$. This provides a theoretical explanation for ADAM's general success while highlighting why it fails in specific high-curvature scenarios where the angle is not "average."

Empirical Performance

The authors evaluated DEAM on standard benchmarks including the ORL Face Dataset, MNIST, and CIFAR-10, covering both convex and non-convex optimization tasks.

Comparison of test loss on ORL and train loss on MNIST across different optimizers.

Key findings include:

  • Faster Convergence: DEAM consistently reached lower loss values in fewer iterations than ADAM, RMSProp, AdaGrad, and SGD across all tested models (Logistic Regression, DNN, and CNN).
  • Wall-Clock Efficiency: On CIFAR-10, DEAM reduced training time by approximately 15% compared to ADAM (57,761s vs. 67,584s), representing significant savings for large-scale models.
  • Path Stability: The combination of adaptive weights and the backtrack term resulted in a more direct path to the minimum, effectively dampening the "zig-zag" behavior seen in other optimizers.

Conclusion

DEAM offers a significant improvement over ADAM by making the momentum weight sensitive to the geometry of the loss landscape. By incorporating a discriminative angle and a backtrack mechanism, it achieves faster, more stable convergence with less manual tuning. While theoretical guarantees for non-convex landscapes remain an area for future formal proof, the empirical results across diverse architectures demonstrate DEAM's potential as a powerful, plug-and-play optimizer for deep learning.

Summary updated

Abstract

Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameters update route. However, there exist errors in some update steps in optimization algorithms with momentum like ADAM. The fixed momentum weight (e.g., \beta_1 in ADAM) will propagate errors in momentum computing. In this paper, we introduce a novel optimization algorithm, namely Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and non-convex situations.
BibTeX
@misc{bai2019deam,
  author = {Jiyang Bai and Jiawei Zhang},
  title = {{DEAM: Accumulated Momentum with Discriminative Weight for Stochastic Optimization}},
  year = {2019},
  eprint = {1907.11307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.11307},
}
Figure