A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)

Stella BidermanarXiv 2023 · DOI · arXiv

Abstract

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce Pythia, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend Pythia to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at https://github.com/EleutherAI/pythia.

Introduction

The transition of a Large Language Model (LLM) from random initialization to a sophisticated reasoning engine remains largely opaque. While scaling laws describe how final performance improves with compute and data, they offer little insight into the internal learning dynamics. Most state-of-the-art models are released only as final weights, preventing researchers from studying the developmental trajectory of these systems.

The Pythia suite addresses this gap by providing a highly controlled "laboratory" for studying LLM development across both training time and model scale. By training multiple models on the exact same data in the exact same order and releasing intermediate checkpoints, Pythia enables scientific inquiry into how behaviors, biases, and capabilities emerge.

The Pythia Suite

The suite consists of 16 models across eight sizes, ranging from 70 million to 12 billion parameters. Two sets were produced: one trained on the standard Pile dataset and another on a deduplicated version.

Table 1: Architectural specifications and equivalent models for the Pythia suite.

Methodology

To ensure scientific consistency, every model size "sees" the same sequence of tokens. The architecture utilizes modern transformer components, including Flash Attention, Rotary Embeddings, and parallel attention/feed-forward layers. Notably, the researchers employed significantly larger batch sizes for smaller models than is standard, achieving a 10x wall-clock speedup without sacrificing performance parity with other suites like OPT.

The release includes 154 checkpoints per model: 144 saved linearly every 1,000 steps and 10 log-spaced checkpoints from the very beginning of training to capture early learning bursts.

Key Insights into Training Dynamics

The controlled nature of Pythia allowed for several novel case studies:

  • Uniform Memorization: Analysis suggests that memorization follows a Poisson process. Contrary to the idea that early or late data is prioritized, memorization appears essentially random and uniform across the training duration.
  • The 2.8B Parameter Threshold: Certain capabilities, such as few-shot arithmetic, only begin to correlate strongly with training data term frequency once a model reaches approximately 2.8B parameters. Below this threshold, models struggle to learn these patterns regardless of exposure frequency.
  • Deduplication Effects: Surprisingly, deduplicating the training data (The Pile) yielded no significant improvement in downstream task performance, challenging common assumptions regarding data quality.

Case Study: Bias Mitigation

One of the most powerful applications of the Pythia suite is the ability to perform "counterfactual" training interventions. Researchers took specific checkpoints and resumed training on modified data—for example, swapping masculine pronouns for feminine ones—to observe the effect on stereotypical bias.

Figure 1: Stereotypical bias levels over training time, showing the impact of debiasing interventions across different model scales.

The results indicated that larger models are often more responsive to these debiasing interventions than smaller ones, suggesting that scale may actually facilitate certain types of safety alignment.

Conclusion

Pythia demonstrates that scientific control in LLM training can yield insights that are impossible to glean from static model releases. By providing the community with intermediate checkpoints and reproducible data loaders, the suite opens new doors for mechanistic interpretability and AI safety research. Future work will leverage these tools to understand how specific neural circuits, such as those responsible for induction or bias, stabilize during training.

Summary updated

Abstract

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at \url{https://github.com/EleutherAI/pythia}.
BibTeX
@misc{biderman2023note,
  author = {Stella Biderman},
  title = {{A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)}},
  year = {2023},
  eprint = {2304.01373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.01373},
}
Figure