GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob SteinhardtarXiv 2021 · DOI · arXiv

Abstract

GPT-Neo is an implementation of model & data-parallel GPT-2 and GPT-3-like models, utilizing Mesh Tensorflow for distributed support. This codebase is designed for TPUs. It should also work on GPUs, though we do not recommend this hardware configuration.
BibTeX
@misc{hendrycks2021gpt,
  author = {Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},
  title = {{GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow}},
  year = {2021},
  eprint = {2105.09938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.09938},
}
Figure