Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning

Jiawu Tian, Liwei Xu, Xiao‐Wei Zhang, Yongqi Li

TL;DR

CG-like-Adam is a novel optimization algorithm that integrates a "rectified" version of the Conjugate Gradient (CG) method into the popular Adam optimizer. By dampening the conjugate term over time, the authors achieve faster convergence and better stability than standard Adam and existing CG-based variants like CoBA.

What Problem Does This Paper Address?

Training deep neural networks is essentially a massive optimization puzzle. While Stochastic Gradient Descent (SGD) is the classic tool, it often struggles with speed and stability. Adam solved many of these issues by using adaptive learning rates, but it isn't perfect—it can fail to converge in certain non-convex scenarios.

On the other hand, Conjugate Gradient (CG) methods are legendary in traditional optimization for their efficiency in solving large-scale problems. However, simply "plugging" CG into deep learning usually leads to disaster (divergence) because the stochastic (noisy) nature of neural network gradients clashes with CG's aggressive direction updates. This paper aims to bridge that gap, creating a version of CG that plays nice with the stochastic, adaptive world of Adam.

Methods

The authors introduce CG-like-Adam, which features two primary innovations:

  1. The "CG-like" Rectification: In vanilla CG, the update direction is a mix of the current gradient and the previous direction, scaled by a coefficient $\gamma_t$. The authors "rectify" this by dividing $\gamma_t$ by $t^a$ (where $t$ is the iteration count and $a \ge 1/2$). This creates a monotonic decreasing sequence that gradually reduces the influence of past directions, stabilizing the algorithm as training progresses.
  2. Dual-Moment Integration: Unlike previous attempts that only use CG for the first-order moment (the "velocity"), CG-like-Adam applies this CG-like direction to both the first-order and second-order moment estimations.
  3. Theoretical Rigor: The paper provides a convergence proof for non-convex objective functions. Crucially, they tackle the "constant coefficient" case for the first-order moment—a scenario that is often a "black box" or an open problem in other adaptive gradient proofs.

Datasets

The algorithm was evaluated on two staple computer vision benchmarks:

  • CIFAR-10: 60,000 images across 10 classes.
  • CIFAR-100: 60,000 images across 100 classes.
  • Models: The researchers used VGG-19 and ResNet-34 to ensure the optimizer performed well across different architectures.

Key Insights and Ideas

  • The $t^a$ "Secret Sauce": The most important "aha moment" is the realization that vanilla CG is too "stiff" for deep learning. By adding the $t^a$ denominator, the authors effectively create a "decaying memory" of past directions. This prevents the optimizer from oscillating wildly due to noisy gradients early in training.
  • Fixing Adam’s Flaws: By incorporating CG directions, the optimizer explores the loss landscape more effectively than Adam’s standard momentum, which can sometimes get stuck or take inefficient paths.
  • Universal Compatibility: The authors tested five different classic CG formulas (Hestenes-Stiefel, Fletcher-Reeves, etc.) within their framework. They found that their "CG-like" modification made almost all of them superior to standard Adam.

Key Results

  • Faster Convergence: In experiments on VGG-19, CG-like-Adam reached 100% training accuracy significantly faster and more stably than both standard Adam and the previous state-of-the-art CG variant, CoBA.
  • Superior Training Loss: Across various learning rates (from $10^{-3}$ down to $10^{-6}$), CG-like-Adam consistently maintained a lower training loss than its competitors.
  • Robustness: The algorithm proved to be less sensitive to the specific choice of the conjugate coefficient formula, making it a more "plug-and-play" solution for researchers.

Limitations and Future Work

  • The Overfitting Trap: On the ResNet-34/CIFAR-100 tests, the algorithm converged so aggressively (hitting 100% training accuracy in under 10 epochs) that it occasionally led to slightly lower test accuracy compared to slower optimizers. This suggests that while it's a faster "engine," it might need better "brakes" (regularization) to prevent overfitting.
  • Hyperparameter Sensitivity: While robust, the algorithm introduces a new parameter ($a$). While the authors suggest $a \ge 1/2$, finding the optimal value for a specific new architecture might require tuning.
  • Future Directions: The authors plan to explore variance reduction techniques to further stabilize the gradients and investigate line-search methods to automatically find the best step size during training.

Summary updated

Abstract

Training deep neural networks is a challenging task. In order to speed up training and enhance the performance of deep neural networks, we rectify the vanilla conjugate gradient as conjugate-gradient-like and incorporate it into the generic Adam, and thus propose a new optimization algorithm named CG-like-Adam for deep learning. Specifically, both the first-order and the second-order moment estimation of generic Adam are replaced by the conjugate-gradient-like. Convergence analysis handles the cases where the exponential moving average coefficient of the first-order moment estimation is constant and the first-order moment estimation is unbiased. Numerical experiments show the superiority of the proposed algorithm based on the CIFAR10/100 dataset.
BibTeX
@misc{tian2024conjugate,
  author = {Jiawu Tian and Liwei Xu and Xiao‐Wei Zhang and Yongqi Li},
  title = {{Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning}},
  year = {2024},
  eprint = {2404.01714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01714},
}
Figure