In modern natural language processing, Byte-Pair Encoding (BPE) is the standard method for breaking text into manageable pieces. By iteratively merging frequent character sequences into subword units, BPE allows models to handle rare words and morphologically complex languages. However, the "standard" settings used for high-resource languages like English often fail when applied to languages with limited data.
Recent research across machine translation, reading comprehension, and sequence tagging suggests a counter-intuitive principle: in low-resource settings, smaller vocabularies are almost always better. By aggressively restricting the number of subword units, researchers can force models to learn dense, frequent representations rather than struggling with sparse, rare tokens.
The Sparsity-Frequency Trade-off
The core challenge in low-resource NLP is data sparsity. If a model uses a large vocabulary (e.g., 32,000 tokens), many of those tokens may only appear once or twice in a small training set. As Aditya Kumar Pathak and colleagues note in their work on Tamil translation, constraining the vocabulary size ensures the model learns dense representations for frequently occurring subwords, preventing it from overfitting to rare "Out of Vocabulary" (OOV) terms.
Biao Zhang and Rico Sennrich demonstrated that by reducing the subword vocabulary to as few as 2,000 symbols, Neural Machine Translation (NMT) can outperform traditional statistical methods even with only 100,000 words of data. This works because finer granularity allows the model to learn the "building blocks" of a language—morphemes and common character clusters—rather than trying to memorize full words that it rarely sees.
Finding the Granularity Sweet Spot
The "granularity" of a tokenizer is determined by the number of merge operations performed during BPE. Zhuosheng Zhang and Hai Zhao describe this as a "sweet spot" between character-level and word-level representations.
- Too few merges: The model acts like a character-level model, losing the semantic context provided by larger units.
- Too many merges: The model reverts to word-level behavior, suffering from the OOV problem and sparse embeddings.
For low-resource languages like Filipino and Cebuano, Kristine Mae M. Adlaon and Nelson Marcos found that the optimal range is typically between 2,000 and 8,000 merge operations. Similarly, Jun Suzuki found that 4,000 units often yielded better performance than the larger vocabularies standard in high-resource NMT.
Subword Fertility and Script Efficiency
A key metric in this optimization is "subword fertility"—the average number of tokens produced per word. Israel Abebe Azime and colleagues found that for Ethiopian languages like Amharic and Tigrinya, tokenizers that produce fewer fragments per word generally lead to better model utility. High fertility (too many fragments) increases the sequence length and the complexity of the task, while low fertility (closer to word-level) risks sparsity.
This sensitivity to granularity isn't just about translation. In the Subword-augmented Embedding (SAW) Reader, Zhuosheng Zhang showed that subword units significantly improve Chinese reading comprehension by capturing morphological nuances that word-level embeddings miss. Even in chemistry, Yuchen Shen and Barnabás Póczos applied this logic to molecular graphs with GraphBPE, discovering that frequent subgraphs (hypernodes) help models generalize better on small molecular datasets.
Language-Specific Challenges
While smaller is often better, the "optimal" size is highly language-dependent. Kevin Duh and Pamela Shapiro argue that BPE and Character-based CNNs are often complementary, particularly for morphologically rich languages. However, they note that Semitic languages like Hebrew remain a unique challenge, as their non-concatenative morphology (where word roots are modified internally) is harder for standard BPE to segment effectively.
In extreme cases, such as isolating functional lexicons (pronouns and prepositions), Zachary William Hopton and colleagues found that as few as 50 merges can be effective. This minimal merging prevents the model from conflating grammatical "glue" with specific lexical contexts, providing a cleaner signal for low-resource tasks like interlinear glossing.
Conclusion
The shift toward smaller subword vocabularies represents a move away from "one-size-fits-all" tokenization. By tuning the number of BPE merges to the specific size and morphological complexity of a dataset, researchers can mitigate the effects of data scarcity. As demonstrated by Jerin Philip and the IIIT Hyderabad team, combining restricted vocabularies with aggressive regularization allows neural models to thrive even in the most data-constrained environments.