Improving First-Order Optimization Algorithms (Student Abstract)

Ange Tato, Roger NkambouAAAI 2020 · DOI

Abstract

First-order optimization algorithms such as Adam and SGD are foundational to training deep neural networks. However, these methods often fail to fully exploit the momentum of the descent path when gradient directions remain consistent. This paper introduces an "Accelerated Algorithm" (AA) wrapper that provides an additional "push" to parameters when current gradients align with previous update directions. Empirical results across MNIST, CIFAR-10, and IMDB datasets demonstrate that AA-enhanced optimizers achieve faster convergence and lower final loss compared to their standard counterparts.

Introduction

The efficiency of deep learning training is largely dictated by the optimization algorithm's ability to navigate complex loss landscapes. While adaptive methods like Adam and AMSGrad are widely used, they typically maintain a steady pace even when the gradient direction is stable over multiple iterations. This research addresses the gap by proposing a heuristic acceleration technique designed to reach the "loss valley" more rapidly without compromising the underlying mathematical properties of the base optimizers.

Methodology: The Accelerated Algorithm (AA)

The proposed AA technique acts as a conditional modifier to standard update rules, inspired by the physical intuition of adding force to a moving object that is already traveling in the desired direction.

Algorithm Logic

At each iteration $t$, the algorithm evaluates the current gradient $g_t$ and the previous moment $m_t$. The acceleration is applied only if two conditions are met:

  1. Directional Alignment: The dot product $g_t \cdot m_t > 0$, indicating the descent is continuing in a similar direction.
  2. Significance Threshold: The magnitude of the change $|m_t - g_t|$ exceeds a user-defined threshold $S$.

If these conditions are satisfied, the standard gradient is replaced by a "boosted" version: $gm_t = g_t + m_t$. This boosted gradient is then passed to the base optimizer (e.g., resulting in AAdam or AAMSGrad). The threshold $S$ serves as a critical safety mechanism, preventing excessive oscillation near local minima where gradients are small.

Experimental Setup

The AA wrapper was evaluated across three distinct architectures and datasets:

  • MNIST: Logistic Regression and Multi-layer Perceptron (MLP).
  • CIFAR-10: Convolutional Neural Networks (CNN) with a c32-c64-c128-c128 architecture.
  • IMDB Movie Reviews: Long Short-Term Memory (LSTM) networks for sentiment analysis.

Results and Discussion

The "Accelerated" variants consistently outperformed standard optimizers in terms of both convergence speed and final loss quality.

Figure 1: Training loss comparison on CIFAR-10 using a CNN. AA-variants (AAdam, AAMSGrad) show significantly faster descent than their base versions.

As seen in Figure 1, AAdam and AAMSGrad exhibit a steeper decline in training loss on the CIFAR-10 dataset compared to standard Adam and AMSGrad. This trend is mirrored in recurrent architectures.

Figure 2: Training loss on IMDB sentiment analysis using an LSTM. The AA-wrapper improves performance across multiple base optimizers, including Adagrad and Adam.

Figure 2 highlights the universality of the approach, showing that the AA wrapper improves the performance of Adagrad, Adam, and AMSGrad on the IMDB LSTM task. In all experiments, AAdam emerged as the top performer, suggesting that the acceleration heuristic synergizes effectively with Adam's adaptive learning rates.

Conclusion and Future Work

The AA wrapper provides a simple yet effective means of accelerating first-order optimization. By leveraging directional consistency, it reduces training time and often finds better minima. However, the method's sensitivity to the threshold parameter $S$ remains a limitation. Future research will explore automated tuning of $S$ and the application of this technique to sparse data environments.

Summary updated

BibTeX
@inproceedings{tato2020improving,
  author = {Ange Tato and Roger Nkambou},
  title = {{Improving First-Order Optimization Algorithms (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7240},
}
Figure