Automatic Learning Rate Maximization in Large Adaptive Machines

Yann LeCun, Patrice Y. Simard, Barak A. Pearlmutter

Abstract

We propose a very simple, and well principled way of computing the optimal step size in gradient descent algorithms. The on-line version is very efficient computationally, and is applicable to large backpropagation networks trained on large data sets. The main ingredient is a technique for estimating the principal eigenvalue(s) and eigenvector(s) of the objective function's second derivative ma(cid:173) trix (Hessian), which does not require to even calculate the Hes(cid:173) sian. Several other applications of this technique are proposed for speeding up learning, or for eliminating useless parameters.
BibTeX
@inproceedings{lecun1992automatic,
  author = {Yann LeCun and Patrice Y. Simard and Barak A. Pearlmutter},
  title = {{Automatic Learning Rate Maximization in Large Adaptive Machines}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
}

Loading related content...

Figure