In the standard pipeline for training a Large Language Model (LLM) or a Machine Translation (MT) system, the choice of vocabulary size is often treated as a historical artifact. Most researchers default to "magic numbers" like 32,000 or 50,000 tokens, inherited from early BERT or T5 configurations. However, a growing body of research suggests that vocabulary size is not just a preprocessing detail, but a critical architectural hyperparameter that governs a model's efficiency, its ability to generalize, and even its tendency to memorize sensitive data.
The central challenge is that finding the "optimal" vocabulary has traditionally been a brute-force endeavor. As Hao Zhou and colleagues note in their work on VOLT, identifying the best subword segmentation typically requires training dozens of model variants to see which one yields the best performance—a process that can consume hundreds of GPU hours.
The Mathematical Bridge: Zipf’s Law and Perplexity
To understand why vocabulary size matters, we must first look at the trade-off between coverage and granularity. If a vocabulary is too small, the model must represent words as long sequences of tiny fragments (like characters), increasing the computational burden on the attention mechanism. If it is too large, the model suffers from "sparse" parameters—many tokens appear so rarely that the model never learns meaningful embeddings for them.
Hayato Kobayashi provided a theoretical foundation for this in Perplexity on Reduced Corpora. By leveraging Zipf’s Law—the observation that word frequency follows a power-law distribution—Kobayashi derived formulae to predict how perplexity degrades as a vocabulary is pruned. This established that the "engineering hack" of removing rare words is actually a mathematically predictable optimization problem.
From Brute Force to Optimal Transport
If we accept that an optimal vocabulary exists, how do we find it without training a hundred models? The VOLT framework, proposed by Hao Zhou and Lei Li, offers an elegant solution by framing vocabulary selection as an Optimal Transport problem.
Instead of trial and error, VOLT treats the search for a vocabulary as a way to maximize "Marginal Utility"—an information-theoretic metric that balances the entropy of the token distribution against the total number of tokens used. By finding the most efficient way to "transport" the raw text into a discrete set of symbols, VOLT can discover high-performing vocabularies in minutes on a CPU, outperforming the expensive grid searches used in previous years.
The Third Dimension of Scaling Laws
For years, the "Scaling Laws" of AI focused almost exclusively on two variables: the number of parameters and the volume of training data. Chaofan Tao and Niklas Muennighoff recently demonstrated in Scaling Laws with Vocabulary that vocabulary size is a critical third dimension.
Their research shows that as models grow in parameter count, they "deserve" larger vocabularies. A massive model has the capacity to learn the nuances of half a million distinct tokens, and doing so actually makes the model more compute-efficient by shortening the sequence lengths it has to process. This is echoed by Sho Takase in Large Vocabulary Size Improves Large Language Models, which argues that the industry-standard 30k–60k range is far too small for modern LLMs, and that expanding to 500k tokens can yield significant reasoning gains.
Task-Specific and Asymmetrical Segmentation
The "one-size-fits-all" approach to vocabulary is also being challenged in specialized domains:
- Machine Translation: Manish Shrivastava and Saumitra Yadav have shown that asymmetrical Byte Pair Encoding (BPE)—where the source language uses a different vocabulary size than the target—is often superior, especially in low-resource settings.
- Multitask Learning: Rather than picking one granularity, Tejas Srinivasan and Florian Metze introduced Block Multitask Learning (BMTL), which trains a single encoder to predict multiple subword granularities simultaneously, making the model more robust to different linguistic structures.
- Chemistry: In molecular generation, Zijie Geng and Yingce Xia applied these concepts to "mine" frequent motifs in chemical libraries, showing that connection-aware vocabularies outperform rigid, human-defined molecular building blocks.
The Hidden Cost: Memorization and Privacy
While larger vocabularies improve performance, they come with a hidden trade-off. Marco Baroni and Dieuwke Hupkes discovered that vocabulary size is a primary driver of memorization.
When a model uses a large vocabulary, it represents the same amount of information using fewer total tokens. This "compression" makes it easier for the Transformer's attention mechanism to map specific input sequences to specific outputs, increasing the risk of the model memorizing training data verbatim. This creates a new frontier for researchers: finding a vocabulary that is large enough for efficiency, but small enough to preserve differential privacy.
By moving away from arbitrary defaults and toward these principled selection methods, the field is transforming tokenization from a "solved" preprocessing step into a dynamic tool for optimizing model intelligence and safety.