TL;DR
CG-like-Adam is a novel optimization algorithm that integrates a "rectified" version of the Conjugate Gradient (CG) method into the popular Adam optimizer. By dampening the conjugate term over time, the authors achieve faster convergence and better stability than standard Adam and existing CG-based variants like CoBA.
What Problem Does This Paper Address?
Training deep neural networks is essentially a massive optimization puzzle. While Stochastic Gradient Descent (SGD) is the classic tool, it often struggles with speed and stability. Adam solved many of these issues by using adaptive learning rates, but it isn't perfect—it can fail to converge in certain non-convex scenarios.
On the other hand, Conjugate Gradient (CG) methods are legendary in traditional optimization for their efficiency in solving large-scale problems. However, simply "plugging" CG into deep learning usually leads to disaster (divergence) because the stochastic (noisy) nature of neural network gradients clashes with CG's aggressive direction updates. This paper aims to bridge that gap, creating a version of CG that plays nice with the stochastic, adaptive world of Adam.
Methods
The authors introduce CG-like-Adam, which features two primary innovations:
- The "CG-like" Rectification: In vanilla CG, the update direction is a mix of the current gradient and the previous direction, scaled by a coefficient $\gamma_t$. The authors "rectify" this by dividing $\gamma_t$ by $t^a$ (where $t$ is the iteration count and $a \ge 1/2$). This creates a monotonic decreasing sequence that gradually reduces the influence of past directions, stabilizing the algorithm as training progresses.
- Dual-Moment Integration: Unlike previous attempts that only use CG for the first-order moment (the "velocity"), CG-like-Adam applies this CG-like direction to both the first-order and second-order moment estimations.
- Theoretical Rigor: The paper provides a convergence proof for non-convex objective functions. Crucially, they tackle the "constant coefficient" case for the first-order moment—a scenario that is often a "black box" or an open problem in other adaptive gradient proofs.
Datasets
The algorithm was evaluated on two staple computer vision benchmarks:
- CIFAR-10: 60,000 images across 10 classes.
- CIFAR-100: 60,000 images across 100 classes.
- Models: The researchers used VGG-19 and ResNet-34 to ensure the optimizer performed well across different architectures.
Key Insights and Ideas
- The $t^a$ "Secret Sauce": The most important "aha moment" is the realization that vanilla CG is too "stiff" for deep learning. By adding the $t^a$ denominator, the authors effectively create a "decaying memory" of past directions. This prevents the optimizer from oscillating wildly due to noisy gradients early in training.
- Fixing Adam’s Flaws: By incorporating CG directions, the optimizer explores the loss landscape more effectively than Adam’s standard momentum, which can sometimes get stuck or take inefficient paths.
- Universal Compatibility: The authors tested five different classic CG formulas (Hestenes-Stiefel, Fletcher-Reeves, etc.) within their framework. They found that their "CG-like" modification made almost all of them superior to standard Adam.
Key Results
- Faster Convergence: In experiments on VGG-19, CG-like-Adam reached 100% training accuracy significantly faster and more stably than both standard Adam and the previous state-of-the-art CG variant, CoBA.
- Superior Training Loss: Across various learning rates (from $10^{-3}$ down to $10^{-6}$), CG-like-Adam consistently maintained a lower training loss than its competitors.
- Robustness: The algorithm proved to be less sensitive to the specific choice of the conjugate coefficient formula, making it a more "plug-and-play" solution for researchers.
Limitations and Future Work
- The Overfitting Trap: On the ResNet-34/CIFAR-100 tests, the algorithm converged so aggressively (hitting 100% training accuracy in under 10 epochs) that it occasionally led to slightly lower test accuracy compared to slower optimizers. This suggests that while it's a faster "engine," it might need better "brakes" (regularization) to prevent overfitting.
- Hyperparameter Sensitivity: While robust, the algorithm introduces a new parameter ($a$). While the authors suggest $a \ge 1/2$, finding the optimal value for a specific new architecture might require tuning.
- Future Directions: The authors plan to explore variance reduction techniques to further stabilize the gradients and investigate line-search methods to automatically find the best step size during training.
Summary updated