GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

Sid Black, Leo Gao, Phil Wang, Connor Leahy, Stella BidermanZenodo 2021 · DOI

Abstract

GPT-Neo is an implementation of model & data-parallel GPT-2 and GPT-3-like models, utilizing Mesh Tensorflow for distributed support. This codebase is designed for TPUs. It should also work on GPUs, though we do not recommend this hardware configuration.
BibTeX
@inproceedings{black2021gpt,
  author = {Sid Black and Leo Gao and Phil Wang and Connor Leahy and Stella Biderman},
  title = {{GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow}},
  booktitle = {Zenodo},
  year = {2021},
  doi = {10.5281/zenodo.5297715},
}

Loading related content...

Figure