A Systematic Evaluation of Large Language Models of Code

Frank F. Xu, Uri Alon, Graham Neubig, Vincent J. HellendoornZenodo 2022 · DOI · OpenAlex

Abstract

These are datasets for the paper: "A Systematic Evaluation of Large Language Models of Code" https://arxiv.org/pdf/2202.13169.pdf The code is available at: https://github.com/VHellendoorn/Code-LMs The file "unseen_test_sets.tar.gz" contains test sets of ~100 files in each of 12 programming languages. These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them. In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model. The file "index.zip" includes an index of the <strong>training set</strong> file paths and commit SHAs. The other files, such as "2-7B-150K.tar", are trained model checkpoints, as explained at https://github.com/VHellendoorn/Code-LMs .
BibTeX
@inproceedings{xu2022systematic,
  author = {Frank F. Xu and Uri Alon and Graham Neubig and Vincent J. Hellendoorn and Vincent Josua Hellendoorn},
  title = {{A Systematic Evaluation of Large Language Models of Code}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.6338014},
}

Loading related content...

Figure