Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law

Yanjin He, Qingkai Zeng, Meng Jiang

TL;DR

This paper proposes a principled method for selecting optimal vocabulary sizes for tokenizers by aligning token frequency distributions with Zipf's Law. They demonstrate that this "Zipf alignment" consistently leads to improved downstream task performance across NLP, genomics, and chemistry, offering a data-driven approach to a previously heuristic-based problem.

What Problem Does This Paper Address?

Tokenization is a crucial first step in many machine learning tasks, especially in natural language processing (NLP). It involves breaking down raw text into smaller units (tokens) that a model can process. The choice of how many unique tokens to include in the vocabulary is a critical hyperparameter. Too small a vocabulary can lead to overly fragmented sequences, making it harder for models to understand meaning. Too large a vocabulary can introduce redundancy, increasing computational costs without significant performance gains.

Historically, selecting the optimal vocabulary size has been an ad-hoc process, often relying on heuristics or dataset-specific choices. Existing metrics for evaluating tokenizers have shown poor correlation with actual downstream task performance, especially when moving beyond traditional NLP to domains like genomics or chemistry. This paper addresses the need for a more robust, principled, and generalizable criterion for determining optimal vocabulary sizes.

Methods

The core of this paper's contribution lies in its proposed method for selecting vocabulary size based on Zipf's Law. Zipf's Law, a well-known linguistic phenomenon, states that the frequency of a word is inversely proportional to its rank in a corpus. The authors hypothesize that token distributions that closely follow this power-law behavior are indicative of optimal tokenization.

To quantify this, they introduce the Zipf alignment score, which measures how closely a tokenizer's token frequency distribution, when plotted on a log-log scale, approximates a straight line (characteristic of a power law). They use the coefficient of determination (R²) as a goodness-of-fit measure for this alignment.

Their approach involves:

  1. Iterative Vocabulary Growth: Starting with a small vocabulary (e.g., all unique characters), they iteratively expand it using a standard algorithm like Byte Pair Encoding (BPE).
  2. Monitoring Zipf Alignment: After each expansion step, they re-tokenize the corpus and calculate the Zipf alignment score (R²).
  3. Optimal Vocabulary Selection: They stop expanding the vocabulary when the Zipf alignment score plateaus, indicating that further increases do not significantly improve the adherence to Zipf's Law. This point is identified as the optimal vocabulary size (Vopt).

This method is designed to be data-driven, adapting to the statistical properties of the specific dataset and domain.

Datasets

The paper uses a variety of datasets across different domains for pre-training and fine-tuning:

  • NLP:
    • Pre-training: OpenWebText, BookCorpus (for BERT); WMT dataset (for mBART).
  • Fine-tuning: GLUE benchmark (for BERT); IWSLT datasets (for mBART translation tasks).
  • Genomics: GUE dataset (for DNA sequence classification).
  • Chemistry: ZINC20 dataset (for SMILES molecular representations); MoleculeNet datasets (BBBP, Tox21, Sider, ClinTox, HIV, BACE for molecular property prediction).

The paper does not introduce new benchmarks but rather applies its method to existing datasets and tasks to demonstrate its generalizability.

Key Insights and Ideas

  • Zipf's Law as a Universal Tokenization Principle: The paper posits that Zipf's Law isn't just for natural language but serves as a fundamental principle for effective tokenization across diverse data types.
  • Automatic Vocabulary Size Selection: The proposed "Zipf alignment score" provides a principled, data-driven way to automatically determine optimal vocabulary sizes, moving away from arbitrary choices.
  • Predictive Power of Zipf Alignment: The Zipf alignment score (R²) is shown to be a strong predictor of downstream task performance, even before training the final model.
  • Cross-Domain Generalizability: The effectiveness of Zipf alignment is demonstrated not only in NLP but also in genomics and chemistry, highlighting its broad applicability.
  • Trade-off Between Fragmentation and Redundancy: The study visually and quantitatively shows how small vocabularies lead to over-fragmentation, while large ones lead to redundancy, and Zipfian alignment helps find the sweet spot.

Key Results

  • NLP: For BERT models on the GLUE benchmark, performance peaked at a vocabulary size of 30,000, which also corresponded to a high Zipf alignment score. Similar trends were observed for mBART translation tasks, where optimal BLEU scores aligned with high R² values.
  • Genomics: Optimal performance was achieved with vocabulary sizes around 4,000, showing a clear correlation between Zipfian goodness-of-fit and accuracy across various classification tasks.
  • Chemistry: For SMILES tokenization, a vocabulary size of 3,000 yielded the best ROC-AUC scores and aligned with the turning point in the R² metric.
  • Correlation: Pearson correlation coefficients between Zipfian goodness-of-fit (R²) and average downstream performance were consistently high (e.g., 0.968 for NLP BERT, 0.954 for Genomics, 0.856 for Chemistry), strongly supporting Hypothesis 2.

Limitations and Future Work

  • Model Scale: The experiments were conducted on relatively small pre-trained models due to hardware limitations. The authors acknowledge that findings might not fully generalize to state-of-the-art, billion-parameter models, and suggest further experiments on larger models.
  • Modality and Architecture Diversity: The study focused on NLP, genomics, and chemistry, and specific model architectures (BERT, mBART). Future work should extend evaluations to other modalities (vision, audio) and diverse model architectures (e.g., Transformers, hybrid models).
  • Further Exploration of Zipfian Behavior: While the paper demonstrates the utility of Zipf's Law, deeper theoretical investigations into why this alignment is so effective across domains could be beneficial.

Summary updated

Abstract

Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model performance. Despite its importance, selecting an optimal vocabulary size remains underexplored, typically relying on heuristics or dataset-specific choices. In this work, we propose a principled method for determining the vocabulary size by analyzing token frequency distributions through Zipf's law. We show that downstream task performance correlates with how closely token distributions follow power-law behavior, and that aligning with Zipfian scaling improves both model efficiency and effectiveness. Extensive experiments across NLP, genomics, and chemistry demonstrate that models consistently achieve peak performance when the token distribution closely adheres to Zipf's law, establishing Zipfian alignment as a robust and generalizable criterion for vocabulary size selection.
BibTeX
@inproceedings{he2025pre,
  author = {Yanjin He and Qingkai Zeng and Meng Jiang},
  title = {{Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure