AUTOSGM: A Unified Lowpass Regularization Framework for Accelerated Learning

Oluwasegun A. Somefun, Stefan Lee, V. John MathewsICASSP 2024 · OpenReview

Abstract

This paper unifies commonly used accelerated stochastic gradient methods (Polyak’s Heavy Ball, Nesterov’s Accelerated Gradient and Adaptive Moment Estimation (Adam)) as specific cases of a general lowpass regularized learning framework, the Automatic Stochastic Gradient Method (AutoSGM). For AutoSGM, we derive an optimal iteration-dependent learning rate function and realize an approximation. Adam is also an approximation of this optimal approach that replaces the iteration-dependent learning-rate with a constant. Empirical results on deep neural networks comparing the learning behavior of AutoSGM equipped with this iteration-dependent learning-rate algorithm demonstrate fast learning behavior, robustness to the initial choice of the learning rate, and can tune an initial constant learning-rate in applications where a good constant learning rate approximation is unknown.
BibTeX
@inproceedings{somefun2024autosgm,
  author = {Oluwasegun A. Somefun and Stefan Lee and V. John Mathews},
  title = {{AUTOSGM: A Unified Lowpass Regularization Framework for Accelerated Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=dHdzbKpPNc},
}

Loading related content...

Figure