Abstract
This paper proposes a principled, domain-agnostic method for selecting the optimal vocabulary size for pre-trained models by aligning token distributions with Zipf’s Law. By measuring how closely a tokenizer’s frequency distribution fits a power-law (the "Zipf alignment score"), we can predict the "sweet spot" where a model achieves peak performance across natural language, genomics, and chemistry.
Introduction
In modern machine learning, tokenization is the bridge between raw data and numerical representations. However, choosing the "right" vocabulary size is currently more of an art than a science. A vocabulary that is too small causes fragmentation, breaking meaningful units into non-semantic pieces. Conversely, an excessively large vocabulary leads to redundancy and sparse data. Most researchers rely on heuristic values (like 32k or 50k) because they worked for previous models. This paper seeks a robust, mathematical criterion to determine the optimal vocabulary size for any sequence-based data.
Methodology: The Zipf Alignment Score
The authors turn to Zipf’s Law, a linguistic observation that the frequency of a word is inversely proportional to its rank in a frequency table.
- Zipf Alignment Score: The token rank-frequency distribution is visualized on a log-log plot. In a perfect Zipfian distribution, this plot is a straight line. This alignment is quantified using the coefficient of determination ($R^2$) of a linear fit.
- The Hypothesis: As vocabulary size increases, the distribution becomes more linear (more Zipfian), and downstream task performance will peak when this alignment is strongest.
- Iterative Selection Strategy: An algorithm gradually expands the vocabulary (using Byte Pair Encoding) and monitors the $R^2$ score. When the score plateaus or "stagnates," the vocabulary has reached its optimal size for that specific dataset.
Experimental Results
The approach was validated across three distinct domains:
- Natural Language (NLP): For BERT-Medium models, optimal performance peaked at a vocabulary size of 30,000 ($R^2 \approx 0.937$). Increasing the size to 50,000 actually led to a slight degradation in the average GLUE score. In translation tasks, BLEU scores tracked the Zipf alignment score almost perfectly.
- Genomics: The optimal vocabulary size was much smaller, around 4,000. At this size, the model achieved superior accuracy across 5 out of 8 genomic tasks in the GUE benchmark.
- Chemistry: The "turning point" for SMILES data was found at a vocabulary size of 3,000, which yielded the highest ROC-AUC scores on MoleculeNet for property prediction.
Key Insights
- Emergent Linearity: As vocabulary size increases, the token distribution naturally "self-organizes" to follow Zipf’s Law. At very small sizes, the log-log plot is curved; as the vocab grows, it straightens out.
- The "Turning Point": Performance does not increase indefinitely. There is a clear inflection point where the $R^2$ alignment stabilizes, consistently correlating with the highest downstream task scores.
- Universal Information Encoding: The applicability to DNA and chemical strings suggests that Zipf’s Law is a fundamental property of efficient information encoding in sequences, not just human language.
- Semantic Preservation: At the "Zipf-optimal" size, the tokenizer captures meaningful units—like functional groups in molecules—whereas sub-optimal sizes break these units apart or merge them inconsistently.
Conclusion
Aligning token distributions with Zipf’s Law provides a robust metric for vocabulary selection. While this study focused on medium-sized models and 1D sequences, the principle offers a promising path toward more scientific model configuration. Future work will investigate if these findings scale to "Large" parameter models and apply to 2D data like image patches or audio spectrograms.
Summary updated