Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learning

Yongqi Li, Xiao‐Wei Zhang

In the world of deep learning, choosing an optimizer is often a tug-of-war between speed and stability. Adam is the go-to for rapid convergence, but it often struggles to generalize as well as good old-fashioned Stochastic Gradient Descent (SGD). Meanwhile, modern architectures use normalization layers (like Batch Norm) that introduce "scale invariance," a property that can cause standard momentum-based optimizers to waste energy growing weight norms rather than minimizing loss.

A new paper from researchers at the University of Electronic Science and Technology of China introduces PadamP, a novel optimization algorithm designed to solve these specific headaches. By combining projection gradients with partially adaptive moment estimation, PadamP aims to give researchers the best of both worlds: the "get-up-and-go" of Adam and the robust generalization of SGD.

TL;DR

PadamP is a new optimizer that prevents unnecessary weight growth in scale-invariant layers by using a dynamic projection technique. It bridges the gap between Adam and SGD using a tunable $p$-th power for moment estimation, resulting in faster convergence and superior test accuracy on standard image benchmarks.

What Problem Does This Paper Address?

The authors tackle two main issues in deep learning optimization:

  1. The Generalization Gap: Adaptive optimizers like Adam often converge quickly but fail to find the "flat minima" that lead to good performance on unseen data.
  2. Radial Weight Growth: In layers with Batch Normalization or Layer Normalization, the direction of the weights matters more than their magnitude (scale invariance). Standard momentum causes weight norms to increase rapidly over time. This "radial growth" effectively shrinks the effective learning rate, slowing down training and potentially hurting performance.

While previous algorithms like AdamP introduced projection techniques to stop this growth, and Padam introduced partial adaptivity, PadamP is the first to integrate these ideas while refining the mathematical conditions for when and how to project the gradient.

Methods

PadamP is built on three key pillars:

  • Partial Adaptivity ($p$-th power): Instead of the standard square root ($p=1/2$) used in Adam to scale gradients, PadamP uses a $p$-th power where $p \in (0, 1/2]$. This allows the algorithm to behave more like SGD (as $p$ approaches 0) or more like Adam (at $p=1/2$), providing a "knob" to tune the adaptivity.
  • Projection Gradient ($\Pi_\theta$): To handle scale-invariant weights, the algorithm projects the update direction onto the tangent space of the weights. This effectively removes the "radial" component of the gradient—the part that only increases the weight's norm without changing the network's output.
  • Dynamic Projection Condition: Unlike its predecessor AdamP, PadamP incorporates the learning rate ($\eta_t$) into its projection criteria. As the learning rate decays, the condition for projecting becomes more stringent. This makes the projection more adaptive to the different stages of training.

The authors also provide a rigorous convergence proof for non-convex problems that "decouples" the momentum coefficients ($\beta_1$ and $\beta_2$), a theoretical hurdle that many previous papers had to bypass with restrictive assumptions.

Datasets

The researchers evaluated PadamP using two classic image classification benchmarks:

  • CIFAR-10: 60,000 images across 10 classes.
  • CIFAR-100: The same number of images but spread across 100 more granular classes.

They tested these datasets using two popular architectures: VGG-16 and ResNet-18.

Key Insights and Ideas

  • The "P" is for Practicality: By including the learning rate in the projection condition, the algorithm automatically realizes that projection is less necessary in the final stages of training when the objective function is near an optimal point.
  • Decoupling Theory from Constraints: Most convergence proofs for Adam-type optimizers require $\beta_1 / \sqrt{\beta_2} < 1$. This paper generalizes the proof, making it applicable to a wider range of real-world hyperparameter settings.
  • The Power of $p$: The authors found that a smaller $p$ (like $1/8$) is particularly effective in the later stages of training, suggesting that "less adaptivity" can actually lead to better stability and lower final loss.

Key Results

The experimental results show that PadamP consistently outperforms Adam, AdamP, and SGDM:

  • Superior Generalization: On VGG-16 (CIFAR-100), PadamP achieved significantly higher test accuracy than Adam and AdamP, proving it finds better-quality solutions.
  • Faster Convergence: PadamP reached 100% training accuracy faster than competing optimizers.
  • Stability: The algorithm showed more stable loss curves, particularly when using a strategy of decaying the $p$-value (e.g., shifting from $p=1/4$ to $p=1/8$ at 100 epochs).
  • Optimal Settings: For VGG-16, $p=1/5$ was found to be the sweet spot, while $p=1/4$ worked best for ResNet-18.

Limitations and Future Work

While PadamP shows great promise, the authors note a few areas for further exploration:

  • Hyperparameter Tuning: While $p$ adds flexibility, it also adds another hyperparameter to tune. The authors suggest an adaptive $p$-decay strategy, but more automated ways to find the optimal $p$ would be beneficial.
  • Scaling Up: The current experiments focus on CIFAR datasets. Testing on massive datasets like ImageNet or in different domains like Natural Language Processing (Transformers) would be the next logical step.
  • Advanced Techniques: The authors suggest that combining PadamP with variance reduction or line-search methods could further enhance its efficiency.

Summary updated

Abstract

Training deep neural networks is challenging. To accelerate training and enhance performance, we propose PadamP, a novel optimization algorithm. PadamP is derived by applying the adaptive estimation of the p-th power of the second-order moments under scale invariance, enhancing projection adaptability by modifying the projection discrimination condition. It is integrated into Adam-type algorithms, accelerating training, boosting performance, and improving generalization in deep learning. Combining projected gradient benefits with adaptive moment estimation, PadamP tackles unconstrained non-convex problems. Convergence for the non-convex case is analyzed, focusing on the decoupling of first-order moment estimation coefficients and second-order moment estimation coefficients. Unlike prior work relying on , our proof generalizes the convergence theorem, enhancing practicality. Experiments using VGG-16 and ResNet-18 on CIFAR-10 and CIFAR-100 show PadamP's effectiveness, with notable performance on CIFAR-10/100, especially for VGG-16. The results demonstrate that PadamP outperforms existing algorithms in terms of convergence speed and generalization ability, making it a valuable addition to the field of deep learning optimization.
BibTeX
@misc{li2025adaptive,
  author = {Yongqi Li and Xiao‐Wei Zhang},
  title = {{Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learning}},
  year = {2025},
  eprint = {2503.10005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10005},
  howpublished = {arXiv},
}
Figure