AdaPID: An Adaptive PID Optimizer for Training Deep Neural Networks
Lists
Only you see this
Abstract
Deep neural networks (DNNs) have well-documented merits in learning nonlinear functions in high-dimensional spaces. Stochastic gradient descent (SGD)-type optimization algorithms are the ‘workhorse’ for training DNNs. Nonetheless, such algorithms often suffer from slow convergence, sizable fluctuations, and abundant local solutions, to name a few. In this context, the present paper draws ideas from adaptive control of dynamical systems, and develops an adaptive proportional-integral-derivative (AdaPID) solver for fast, stable, and effective training of DNNs. AdaPID relies on second-order moment estimates of gradients to adaptively adjust the PID coefficients. Numerical tests corroborate the merits of AdaPID on several tasks such as image generation using generative adversarial networks (GANs) and image classification using convolutional neural networks (CNNs) as well as long-short term memories (LSTMs).
BibTeX
@inproceedings{weng2022adapid,
author = {Boxi Weng and Jian Sun and Alireza Sadeghi and Gang Wang},
title = {{AdaPID: An Adaptive PID Optimizer for Training Deep Neural Networks}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2022},
url = {https://openreview.net/forum?id=zD0a6BIxoxU},
}Loading related content...