Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah A. Smith

Abstract

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah A. Smith. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
BibTeX
@inproceedings{merrill2021effects,
  author = {William Merrill and Vivek Ramanujan and Yoav Goldberg and Roy Schwartz and Noah A. Smith},
  title = {{Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.133},
}
Figure