Abstract: Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training.
Introduction: The Tokenization Budget
Multilingual Language Models (MLMs) like mBERT or XLM-R rely on a shared vocabulary to represent a vast array of languages. However, because vocabulary size is fixed (often around 120k tokens), a "budgeting" problem arises. If the vocabulary allocation is skewed toward a few dominant languages, others are forced into extreme fragmentation, breaking words into tiny, less meaningful sub-units like individual characters. This paper investigates how the allocation of these limited "slots" and the overlap of tokens across languages dictate model performance.
Quantifying Tokenizer Properties
To move beyond qualitative assessments, we propose new non-parametric measures to evaluate tokenizer behavior:
- Allocation Metrics: We use Average Rank (AR) and Characters Per Token (CPT) to measure fragmentation. A higher CPT indicates the tokenizer captures longer, more semantically rich chunks for a given language.
- Overlap Metric: We employ Jensen-Shannon Divergence (JSD) to quantify the distance between token distributions across language pairs. Low JSD indicates high vocabulary sharing.
We tested these metrics by training 150M-parameter Transformer models from scratch using four distinct tokenization strategies: standard BPE and Unigram LM, a novel TOKMIX method (which balances monolingual tokenizers), and a NOOVERLAP control where languages are forced to use unique token sets.
The Overlap Paradox
A common assumption in multilingual modeling is that sharing tokens across languages facilitates "cross-lingual magic." Our results reveal a more nuanced reality. While high overlap significantly benefits Named Entity Recognition (NER) and Natural Language Inference (NLI), it acts as a hindrance for syntactic tasks like Part-of-Speech (POS) tagging and Dependency Labeling.
This "Overlap Paradox" stems from ambiguity. Shared tokens often represent different grammatical functions in different languages (e.g., the token "a" serving as an article in English but a preposition in Spanish). For word-level precision, this cross-lingual interference can degrade the model's ability to resolve local syntactic structures.
Key Results and the Task Quadrant
Our analysis demonstrates that tasks cluster based on their sensitivity to allocation and overlap. We visualized this relationship in a "Quadrant Map," which serves as a guide for selecting tokenizers based on the intended application.
- Sentence-level tasks (Retrieval, NLI) thrive in the "High Overlap" zone.
- Word-level tasks (POS, Dependency Labeling) require "High Allocation" to succeed, as they depend on the model's ability to recognize longer, language-specific morphological units.
- BPE vs. Unigram: In multilingual settings, BPE consistently allocates vocabulary more effectively than Unigram, leading to superior performance in diverse language tests (e.g., a POS tagging F1 of 67.3 for BPE vs. 54.0 for Unigram).
Conclusion and Practical Implications
One of the most striking findings is that standard training objectives, like Masked Language Modeling (MLM) scores, can be deceptive. Models with high vocabulary allocation often show lower MLM scores during pre-training but achieve significantly higher downstream performance.
For researchers and developers, these findings suggest that tokenization should not be an afterthought. By auditing a tokenizer’s allocation and overlap metrics before pre-training, developers can ensure that low-resource languages are not "starved" of vocabulary space and that the tokenizer is optimized for the specific needs of the target downstream tasks.
Summary updated