Implicitly regularized interaction between SGD and the loss landscape geometry

Sungyoon Lee, Cheongjae JangNeurIPS 2022 · OpenReview

Abstract

We study unstable dynamics of stochastic gradient descent (SGD) and its impact on generalization in neural networks. We find that SGD induces an implicit regularization on the interaction between the gradient distribution and the loss landscape geometry. Moreover, based on the analysis of a concentration measure of the batch gradient, we propose a more accurate scaling rule, Linear and Saturation Scaling Rule (LSSR), between batch size and learning rate.
BibTeX
@inproceedings{lee2022implicitly,
  author = {Sungyoon Lee and Cheongjae Jang},
  title = {{Implicitly regularized interaction between SGD and the loss landscape geometry}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=KHoV9zn1jLE},
}

Loading related content...

Figure