Abstract
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe GPT-NeoX-20B's architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
Introduction
In recent years, Large Language Models (LLMs) have followed a predictable "power law" of scaling: more parameters and more data generally lead to better performance. However, as models have grown to hundreds of billions of parameters, they have increasingly become the "protected intellectual property" of large corporations. This creates a significant barrier for the broader research community, as gated APIs prevent researchers from probing a model’s inner workings, studying its failure modes, or developing safety interventions.
GPT-NeoX-20B addresses this "researcher divide" by providing a high-performing, dense model that is large enough to exhibit complex emergent behaviors but open enough for anyone to study. By releasing the weights, code, and training checkpoints under a permissive license, EleutherAI provides a powerful platform for studying AI safety, interpretability, and the emergent capabilities of scale.
Architecture and Training
GPT-NeoX-20B is a Transformer-based decoder model. While it largely follows the GPT-3 architecture, several key technical deviations were introduced to improve efficiency and performance:
- Rotary Positional Embeddings (RoPE): Instead of standard learned positional embeddings, the model uses RoPE, which encodes relative position by rotating the embedding space. This approach helps the model better handle varying sequence lengths and improves generalization. RoPE rotates query and key vectors to encode relative position. Source, p. 2
- Parallel Attention + Feed-Forward Layers: Standard Transformers compute attention and feed-forward layers in series. GPT-NeoX-20B computes them in parallel and sums the results, which increased training throughput by approximately 15%.
- All Dense Layers: Unlike GPT-3, which uses alternating dense and sparse layers, this model uses exclusively dense layers to reduce implementation complexity.
The model was trained on The Pile, a massive 825 GiB curated dataset designed specifically for training LLMs. It includes 22 diverse data sources, ranging from academic papers (arXiv, PubMed) and code (GitHub) to web scrapes and dialogue. Training was conducted using the DeepSpeed and Megatron libraries on a cluster of 96 NVIDIA A100 GPUs, utilizing a sophisticated pipeline and tensor parallelism setup to maintain high efficiency across nodes.
Key Insights and Results
Tokenizer Efficiency
The authors developed a custom BPE-based tokenizer optimized for The Pile. It is particularly efficient at handling whitespace and code, representing the same text with roughly 10% fewer tokens than the GPT-2 tokenizer. This efficiency allows for a larger effective context window when processing structured data like source code.
Training Beyond One Epoch
Standard practice in LLM training often suggests training for only one epoch to avoid overfitting. However, the authors trained on duplicated data for more than one epoch and found no evidence of performance degradation. As shown in the generalization error plot, validation loss continued to fall even after crossing the one-epoch boundary, suggesting that the "one epoch" rule may be less rigid than previously thought. Generalization error plot showing validation loss continued to fall after crossing the one-epoch boundary. Source, p. 6
Benchmark Performance
To evaluate the model, the authors used the EleutherAI Language Model Evaluation Harness, comparing GPT-NeoX-20B against GPT-3, GPT-J-6B, and various FairSeq models.
- Broad Competitiveness: GPT-NeoX-20B outperformed the FairSeq 13B model on 22 out of 32 evaluation tasks.
- Mathematical Reasoning: The model showed particularly strong performance on arithmetic and mathematical tasks, consistently outperforming GPT-J and FairSeq models.
- Few-Shot Learning: One of the most striking findings is that GPT-NeoX-20B is an exceptionally strong few-shot reasoner. In five-shot evaluations on the MMLU benchmark, it showed a massive improvement over its zero-shot performance, highlighting its ability to learn effectively from context.
Limitations and Environmental Impact
Despite its strengths, GPT-NeoX-20B has limitations. Due to the extreme cost of training, a full hyperparameter sweep was not possible; instead, values were interpolated from existing literature. Additionally, while the tokenizer and dataset were designed with code in mind, resource constraints prevented the execution of standard coding benchmarks like HumanEval.
Finally, the authors transparently documented the model's carbon footprint, estimating that training emitted approximately 31.73 metric tons of CO2. They advocate for more systematic reporting of these metrics to better understand the environmental costs of large-scale AI research.
Summary updated