Appropriate Learning Rates of Adaptive Learning Rate Optimization\n Algorithms for Training Deep Neural Networks

Hideaki Iiduka

Abstract

This paper deals with nonconvex stochastic optimization problems in deep learning and provides appropriate learning rates with which adaptive learning rate optimization algorithms, such as Adam and AMSGrad, can approximate a stationary point of the problem. In particular, constant and diminishing learning rates are provided to approximate a stationary point of the problem. Our results also guarantee that the adaptive learning rate optimization algorithms can approximate global minimizers of convex stochastic optimization problems. The adaptive learning rate optimization algorithms are examined in numerical experiments on text and image classification. The experiments show that the algorithms with constant learning rates perform better than ones with diminishing learning rates.

Introduction

In the world of deep learning, optimizers like Adam and AMSGrad are the "gold standard." However, there has long been a disconnect between theory and practice. Most theoretical guarantees for these algorithms assume the objective function is convex, whereas deep neural networks are notoriously nonconvex. Furthermore, traditional proofs often require the learning rate to diminish toward zero to ensure convergence. In practice, if the learning rate drops too fast, the model often stops learning entirely before reaching a high-quality solution.

This research bridges this gap by providing a theoretical foundation for using constant learning rates in adaptive optimization algorithms for nonconvex problems. By shifting the goal from global regret minimization to finding "stationary points"—places where the gradient is zero—the paper provides a more realistic and useful theoretical framework for deep learning practitioners.

Methods and Mathematical Framework

The author introduces Algorithm 1, a generalized framework that encompasses both Adam and AMSGrad. The key methodological contributions include a unified mathematical analysis that handles both constant and diminishing "sub-learning rates" ($\alpha_n$ and $\beta_n$).

To satisfy theoretical requirements for convergence, the author uses a slightly modified version of the Adam/AMSGrad update rules, specifically regarding how the "preconditioner" matrix $H_n$ is defined. This ensures the algorithm remains well-behaved throughout training. The notation used to define these spaces and operators is summarized below.

Table 1: Summary of notation used in the theoretical analysis.

The paper provides rigorous proofs (Theorems 3.1 and 3.2) showing that under certain conditions, these algorithms will converge to a stationary point of a nonconvex function. This shift in focus to stationary points allows for a more honest theoretical backing for what practitioners are actually doing in the field.

Empirical Evaluation

The theoretical findings were tested using two standard benchmarks to evaluate performance on established, difficult tasks:

  • IMDb Dataset: Used for text classification with a Long Short-Term Memory (LSTM) network.
  • CIFAR-10: Used for image classification with a 20-layer Residual Network (ResNet-20).

Key Results

The empirical results were clear and consistent across both text and image tasks:

  1. Superiority of Constant Rates: Algorithms using constant sub-learning rates significantly outperformed those with diminishing rates.
  2. Training Stability: In the CIFAR-10 experiments, the diminishing rate versions often "stalled" early. While constant-rate Adam variants reached high accuracy, diminishing-rate versions struggled to break past baseline levels because their step sizes became too small too soon.
  3. Validation of Theory: The experiments confirmed the "lim sup" bounds predicted in the theoretical analysis, showing that even with a constant rate, the algorithm stays within a predictable, small neighborhood of a stationary point.

Discussion and Limitations

A major takeaway is that constant rates are more "implementable" in complex tasks because they allow the model to keep exploring the landscape, whereas diminishing rates often become approximately zero too quickly. However, some limitations remain:

  • Global vs. Local: While the paper proves convergence to a stationary point, it cannot guarantee that the algorithm won't get stuck in a "bad" local minimum or a saddle point.
  • Hyperparameter Sensitivity: Results still showed sensitivity to the specific constant chosen (e.g., $10^{-2}$ vs $10^{-3}$).
  • Scope: The study focused on LSTMs and ResNets. Future work could explore if these findings hold for modern architectures like Transformers or in domains like Reinforcement Learning.

In conclusion, this work provides the necessary theoretical rigor to justify the use of constant learning rates, aligning mathematical theory with the practical success of adaptive optimizers in deep learning.

Summary updated

Abstract

This paper deals with nonconvex stochastic optimization problems in deep\nlearning and provides appropriate learning rates with which adaptive learning\nrate optimization algorithms, such as Adam and AMSGrad, can approximate a\nstationary point of the problem. In particular, constant and diminishing\nlearning rates are provided to approximate a stationary point of the problem.\nOur results also guarantee that the adaptive learning rate optimization\nalgorithms can approximate global minimizers of convex stochastic optimization\nproblems. The adaptive learning rate optimization algorithms are examined in\nnumerical experiments on text and image classification. The experiments show\nthat the algorithms with constant learning rates perform better than ones with\ndiminishing learning rates.\n
BibTeX
@article{iiduka2022appropriate,
  author = {Hideaki Iiduka},
  title = {{Appropriate Learning Rates of Adaptive Learning Rate Optimization\textbackslash\{\}n Algorithms for Training Deep Neural Networks}},
  journal = {IEEE Trans. Cybernetics},
  year = {2022},
  doi = {10.1109/tcyb.2021.3107415},
}
Figure