Multilingual language models carry vocabularies of a quarter-million tokens or more, and the table that maps each token to a vector can hold most of the model's parameters. When such a model is deployed for one language, most of that table is never used. Vocabulary trimming deletes the unused rows and can roughly halve memory at almost no cost. The catch is how anyone knows the cost is "almost none." Every published check uses labeled benchmarks in a handful of well-resourced languages. Most languages a trimmed model might serve have no labeled test set at all, and those are exactly the languages where trimming is most likely to hurt: their words are rare, so they are the first to lose their dedicated tokens and get split into smaller pieces.
We propose to decide how far to trim using only unlabeled text in the target language and the original, untrimmed model. The central quantity is drift: on held-out raw sentences, how far the trimmed model's internal representations move away from the full model's. We will test whether drift predicts downstream damage well enough to set a per-language trimming budget, validating it on languages that do have labels (African NER, Indigenous American inference, grammatical minimal pairs in 101 languages). We then use the same signal to choose which tokens to keep. The question has a clear answer either way. If drift works, trimming can be certified for hundreds of unbenchmarked languages. If a much cheaper count, such as how often trimming changes the tokenization of held-out text, predicts just as well, we will report that and recommend it.
Core Claim
Trimming changes a model's behavior only through one channel: words whose tokenization changes because one of their tokens was removed. Those words fall back to shorter pieces. Whether that hurts depends on the language and the model, not only on how often it happens. Removing a token that crosses a morpheme boundary may be harmless or even helpful, which fits BPE-knockout's finding that pruning such merges improved Dutch NER, and the French question-answering gain reported for trimming in Ushio et al.. Removing a frequent stem token in an agglutinative language may be damaging.
Our claim is that representation drift, measured against the untrimmed model on unlabeled held-out text, captures this language-specific cost. We expect it to predict labeled downstream loss better than tokenization-change rate, fertility increase (how many more pieces each word is split into), or absolute intelligibility probes. We also claim that a single drift threshold, calibrated on labeled languages, transfers to held-out languages as a stopping rule.
Concretely, for held-out sentences $x$ from a target-language corpus, let $h_{\text{full}}(x)$ and $h_{\text{trim}}(x)$ be final-layer representations pooled per whitespace word, so they are comparable despite different tokenizations. Drift is
$$D = 1 - \mathbb{E}_{x}\,\mathbb{E}_{w \in x}\cos\big(h_{\text{full}}(w), h_{\text{trim}}(w)\big),$$
where $w$ ranges over words in $x$. Words whose tokenization is unchanged still contribute, because context shifts when their neighbors are re-split.
Research Program
Measurement. The first component is a measurement study. We take XLM-R base, an encoder whose 250k-token vocabulary dominates its parameter count, and BLOOM-560M, a decoder on which Bogoychev et al. already documented trimming's memory savings. We trim each model for every language in our labeled test pool at a sweep of vocabulary budgets, using three existing selection rules:
- Corpus occurrence, which keeps every token seen in a target-language sample, as in Ushio et al. and in the vocabulary step of BISTILLATION.
- Top-$n$ frequency, which keeps only the $n$ most frequent tokens in that sample.
- Unicode script filtering, which keeps only tokens written in the target language's script.
Trimming text comes from MADLAD-400, chosen because it covers 419 languages. A disjoint held-out slice is reserved for computing label-free signals. For every combination of model, language, rule and budget, we record the labeled downstream loss relative to the untrimmed model, together with four label-free signals:
- drift $D$;
- the fraction of held-out words whose tokenization changed;
- the increase in fertility;
- the Monolingual Local Sensitivity score from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes, which tests whether the model can match sentences to versions with adjacent characters swapped.
That probe correlates at $r = 0.99$ with downstream accuracy across 44 languages and marks a hard failure threshold at 0.99. We use it as a floor check, not as the main signal, for reasons given below.
Method. The second component uses drift to choose tokens. For each candidate token $t$, we remove $t$ alone, re-tokenize the held-out words that contained it, and measure the average per-word drift $\delta(t)$ on those words. The token's retention score is $s(t) = n(t)\,\delta(t)$, where $n(t)$ is its count in the trimming corpus. Tokens are removed in ascending order of $s$ until either the budget or the calibrated drift threshold is reached.
The cost is manageable. Only tokens that actually occur in the target corpus are candidates, typically tens of thousands. Each evaluation touches only the sentences containing that token. This rule keeps English loanwords, digits and entity names that the target text actually uses, which is where script filtering is known to fail Bogoychev et al..
Calibration and transfer. The third component turns drift into a stopping rule. We fit a threshold on half the labeled languages so that predicted downstream loss stays under one point. We then apply that threshold, unchanged, to the other half and to languages that have no labels.
What we do not claim. We report memory, not GPU latency. Bogoychev et al. found that trimming did not speed up GPU inference, because GPU matrix-multiply kernels are tuned for large, contiguous matrices.
Baselines
Untrimmed model. The naive reference point is the full model, fine-tuned identically. Every loss is measured against it, and drift is defined relative to it.
Corpus-occurrence trimming. The strongest existing method is corpus-occurrence trimming as in Efficient Multilingual Language Model Compression through Vocabulary Trimming. On seven high-resource languages it reports compression of up to 55% for XLM-R on sentiment and 35% for mT5 on generation "within 0.5% of the full versions." It is the strongest competitor because it is lossless by construction for any word covered by the corpus. The real question is what it costs when the corpus is small or off-domain, and how far one can go beyond it. We run it on the same MADLAD-400 samples, with trimming applied after fine-tuning, and the same fine-tuning seeds.
There is no reported result on Lacuna for any vocabulary-trimming method on MasakhaNER, AmericasNLI or MultiBLiMP. BISTILLATION covers such languages but combines vocabulary reduction with layer dropping and reports only a 1–3 point overall gap. We therefore establish the occurrence-trimming numbers ourselves.
Top-$n$ frequency and script filtering. These are the budget-controlled heuristics. Top-$n$ shows the performance cliffs reported by Ushio et al. Script filtering shows the miss rates reported by Bogoychev et al.: generated outputs diverged from the full model's 10–20% of the time for Bulgarian and 30–40% for Chinese on their 50 OpenAssistant prompts. We rerun those prompts on BLOOM-560M as a sanity reproduction.
Matching the comparison. All pruning rules are compared at matched vocabulary sizes, with identical trimming corpora and no additional training. Drift-guided selection counts as beating frequency if it pushes the one-point-loss cliff to a smaller vocabulary on most languages, and does so with non-overlapping seed intervals.
Ablations. The key ablation swaps drift for tokenization-change rate, both as the stopping signal and as the retention score. With $\delta(t)$ replaced by a constant, the retention score reduces to frequency, which directly isolates whether measuring the model's own response adds anything beyond counting.
Why not just…?
Why not just keep every token that appears in a large target-language corpus? For many languages this may already be nearly lossless, and we take that possibility seriously. It gives no control over the budget, though: compression is whatever the corpus happens to leave. For a low-resource language, a small corpus drops valid rare tokens, and the cost is invisible because there is no test set. Our measurement study will show directly how often occurrence trimming is already safe. If it is safe for nearly all languages at acceptable compression, the practical contribution narrows to certifying that fact, and the stopping rule matters only for tighter memory budgets.
Why not just count how much held-out text gets re-tokenized? This is the real threat. It is cheaper than drift and needs no model forward passes. Its weakness is that it treats every changed split as equally costly. Evidence from BPE-knockout and Ushio et al. says some splits cost nothing or help. The Italian vocabulary-adaptation study, Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation, and Ahuja et al. tie harm to fragmentation, but they did not separate how often splits change from how much each split costs. The rank-correlation and threshold-transfer experiments settle this. We commit in advance: if change rate predicts labeled loss within 0.05 Spearman of drift and transfers its threshold equally well, we recommend change rate.
Why not just run the local-structure probe after trimming? This is the obvious combination of the two directions, and we include it. Its limitation is that it only detects one kind of failure. A language scoring below 0.99 is essentially unusable, but scores above 0.99 do not tell you whether the model understands meaning. A trimmed model can therefore lose several NER points while staying far above the threshold. The probe also measures whether the model handles the language at all, not how much trimming changed it. We expect it to catch only catastrophic trims, and the correlation analysis will show whether that is right.
Why not just evaluate on machine-translated benchmarks? Translation adds errors and shifts question difficulty, as the evaluation-gap literature documents. For Indigenous languages, translation quality is low AmericasNLI. A trim-evaluation signal built on translated text would mix up translation artifacts with trimming damage.
Proposed Experiments
Labeled test beds. The primary low-resource test bed is MasakhaNER, with XLM-R fine-tuned per language; untrimmed XLM-R baselines there reach average F1 of about 80–83. It suits this study because its named entities are often in Latin script and borrowed, so it naturally tests code-mixed input.
We also use:
- XNLI, for continuity with prior trimming results.
- AmericasNLI, in the translate-train setting (reported average 49.12%). Zero-shot XLM-R sits near the 33.3% chance level, which would hide any trimming loss.
- MultiBLiMP 1.0, which tests subject-verb agreement across 101 languages; the best models there reach about 90%. We score it on BLOOM-560M by comparing the two sentences' log-likelihoods.
- FLORES-200 byte-normalized perplexity, following Efficient Low-Resource Language Models Using Tokenizer Transfer. Normalizing by bytes makes models with different vocabularies comparable.
Decisive checks. Five results decide the study:
- The Spearman correlation, across all (language, rule, budget) cells, between each label-free signal and labeled loss. The main claim requires drift to clearly outperform change rate and the local-structure probe.
- Threshold transfer to held-out languages, measured as the fraction of held-out languages whose true loss exceeds one point when trimmed to the calibrated drift threshold. We target under 15%.
- The position of the one-point-loss cliff for drift-guided selection versus top-$n$ frequency, at matched vocabulary size.
- Memory saved at the calibrated threshold, reported per language family, so that savings on Latin-script languages do not mask losses elsewhere.
- Whether drift and MultiBLiMP disagree. A trimmed model could drift little yet lose agreement accuracy. That would show drift misses grammatical damage, and the stopping rule would then need a grammatical term.
Output. Finally, we apply the calibrated rule to MADLAD-400 languages with no benchmark and publish per-language trimmed vocabularies with their drift certificates.
References
- BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision
- Efficient Multilingual Language Model Compression through Vocabulary Trimming
- The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics
- Distilling Efficient Language-Specific Models for Cross-Lingual Transfer
- MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
- Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
- AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
- MasakhaNER: Named Entity Recognition for African Languages
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
- Efficient Low-Resource Language Models Using Tokenizer Transfer