On the Role of Momentum in the Implicit Bias of Gradient Descent for Diagonal Linear Networks

Bochen Lyu, Zhanxing ZhuOpenReview 2024 · Submission to ICLR 2024 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Momentum is a widely adopted and crucial modification to gradient descent when training modern deep neural networks. In this paper, we target on the regularization effect of momentum-based methods in regression settings and analyze a popular proxy model, diagonal linear networks, to precisely characterize the implicit bias of heavy-ball (HB) and Nesterov's method of accelerated gradients (NAG). We show that, HB and NAG exhibit different implicit bias compared to GD for diagonal linear networks, which is different from the one for classic linear regression problem where momentum-based methods share the same implicit bias with GD. Specifically, the role of momentum in the implicit bias of GD is twofold. On one hand, HB and NAG induce extra initialization mitigation effects similar to SGD that are beneficial for generalization of sparse regression. On the other hand, besides the initialization of parameters, the implicit regularization effects of HB and NAG also depend on the initialization of gradients explicitly, which may not be benign for generalization. As a consequence, whether HB and NAG have better generalization properties than GD jointly depends on the aforementioned twofold effects determined by various parameters such as learning rate, momentum factor, data matrix, and integral of gradients. Particularly, the difference between the implicit bias of GD and that of HB and NAG disappears for small learning rate. Our findings highlight the potential beneficial role of momentum and can help understand its advantages in practice from the perspective of generalization.
BibTeX
@misc{lyu2024role,
  author = {Bochen Lyu and Zhanxing Zhu},
  title = {{On the Role of Momentum in the Implicit Bias of Gradient Descent for Diagonal Linear Networks}},
  year = {2024},
  url = {https://openreview.net/forum?id=014CgNPAGy},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure