Abstract
Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training.
Introduction
In the development of Large Language Models (LLMs), tokenization is frequently treated as a static pre-processing step. However, in multilingual contexts, the tokenizer acts as the primary architect of the model's internal representation. Because multilingual models must accommodate dozens or hundreds of languages within a fixed-size vocabulary, a "tug-of-war" for lexical space occurs. This paper investigates how the configuration of this space—specifically through Vocabulary Allocation and Vocabulary Overlap—dictates performance across diverse linguistic tasks.
Methodology and Metrics
To evaluate the impact of tokenization, we compare two standard algorithms, Byte-Pair Encoding (BPE) and Unigram LM, against two novel merging strategies designed to isolate the effects of sharing:
- NOOVERLAP: Tokenizers are trained on monolingual corpora independently and then concatenated. This forces zero vocabulary sharing (excluding special tokens), ensuring that each language has its own dedicated lexical units.
- TOKMIX: Monolingual Unigram tokenizers are trained separately, and their probabilities are averaged and trimmed to create a balanced multilingual vocabulary that allows for controlled sharing.
We introduce three primary metrics to quantify tokenizer quality:
- Characters Per Token (CPT): A measure of text fragmentation. Higher CPT indicates the tokenizer captures longer, more meaningful units (Allocation).
- Average Rank (AR): Measures how many vocabulary units are typically "active" for a specific language within the shared space.
- Jensen-Shannon Divergence (JSD): Used to quantify the "distance" or overlap between the token distributions of different languages.
Experimental Setup
Our models were trained on a 10% subset of the CC-100 corpus, covering an initial set of six diverse languages (Arabic, Chinese, Greek, Turkish, Spanish, and English), later expanded to 20. Evaluation was conducted using the XTREME benchmark, categorized into:
- Word-level tasks: Part-of-Speech (POS) tagging, Named Entity Recognition (NER), and Dependency Labeling.
- Sentence-level tasks: Natural Language Inference (NLI) and Cross-lingual Sentence Retrieval (Tatoeba).
To isolate the impact of the representations, we utilized "probing"—training a simple linear layer on top of a frozen model—rather than full fine-tuning.
Key Findings
The Overlap Paradox
Common wisdom in multilingual modeling suggests that maximizing vocabulary overlap facilitates cross-lingual transfer. Our results challenge this assumption, revealing a "double-edged sword" effect. While high overlap acts as a bridge for semantic tasks (NLI, Retrieval) and NER, it introduces significant noise for syntactic tasks. For POS tagging and dependency labeling, having language-specific, non-overlapping tokens proved superior.
Allocation vs. Overlap
For word-level tasks, the "room" a language is allocated in the vocabulary (Allocation) is a more significant predictor of performance than the degree of sharing (Overlap). We found a strong positive correlation ($r = 0.724$) between Characters Per Token (CPT) and POS tagging performance. This suggests that reducing fragmentation is critical for the model to grasp language-specific structural nuances.
The Deception of MLM Loss
Our study found that models achieving the best scores on Masked Language Modeling (MLM)—the standard pre-training objective—often performed worse on downstream tasks. A tokenizer that makes the "fill-in-the-blank" task easy may do so by over-fragmenting words into common sub-units, which ultimately hinders the model's ability to learn deep linguistic features.
Performance Benchmarks
In the 6-language setting, the NOOVERLAP model achieved the highest F1 scores for POS tagging (69.2) and Dependency Labeling (58.8), outperforming standard Unigram and BPE. This underscores the value of language-specific lexical space for syntactic accuracy. Furthermore, we note that in architectures like XLM-RoBERTa, approximately 70% of the parameters reside in the input embedding layer, making the tokenizer's efficiency the primary driver of parameter utilization.
Conclusion
Tokenization is not merely a pre-processing step but a fundamental constraint on multilingual model capability. While BPE generally allocates more meaningful (longer) tokens than vanilla Unigram, the choice of tokenizer should be task-dependent. Developers focusing on syntactic accuracy should prioritize vocabulary allocation and minimize overlap, while those focused on semantic transfer should lean toward shared representations. Future research should explore hybrid strategies that share semantic roots while maintaining distinct syntactic markers.
Summary updated