Vocabulary Adaptation for Efficient Cross Lingual Language Models

20 concepts15 papers

When a state-of-the-art Large Language Model (LLM) processes non-English text, it often encounters an invisible barrier known as the "English tax." Because most prominent LLMs are trained predominantly on English data, their tokenizers are highly optimized for English words but deeply inefficient for other languages. When processing non-Latin scripts or low-resource languages, these tokenizers overfragment text, breaking single words down into long sequences of tiny subwords or raw bytes.

This overfragmentation causes a massive spike in token fertility—the average number of tokens required to represent a word. Because transformer computational overhead and attention mechanisms scale quadratically with sequence length, high token fertility directly translates to slower inference speeds, higher computational costs, and a severely restricted effective context window.

To solve this, researchers have developed techniques for cross-lingual vocabulary adaptation. By modifying or replacing a model's tokenizer to better suit a target language, we can dramatically lower token fertility, unlocking massive efficiency gains without rebuilding models from scratch.

The Two Paths: Extension vs. Replacement

There are two primary strategies for adapting a model's vocabulary to a new language: expanding the existing tokenizer or completely replacing it.

Vocabulary Extension

In this paradigm, researchers keep the model's original tokenizer intact but append a new set of language-specific tokens (typically 10,000 to 32,000) to the vocabulary. Atula Tejaswi et al. demonstrate in Exploring Design Choices for Building Language-Specific LLMs that adding these targeted tokens significantly lowers the token-to-word ratio for non-Latin scripts. This allows English-centric models to achieve throughput levels comparable to native-language models without requiring an excessively large vocabulary.

This mechanism has been successfully deployed across a wide variety of languages to prevent scripts from decomposing into inefficient byte sequences (Hui Lv et al.). Examples include adapting models to Arabic (M Saiful Bari et al.), Bangla (Shahriar Kabir Nahin et al.), Tibetan (Hui Lv et al.), and Japanese (Kazuki Fujii et al.; Hiroyuki Deguchi et al.).

Tokenizer Replacement (Swapping)

Instead of growing the vocabulary, this approach completely discards the original, bulky tokenizer and swaps it for a streamlined, language-specific or family-specific version. Konstantin Dobler et al. show in Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough that tokenizer swapping optimizes the embedding layer and reduces fertility even more effectively than vocabulary extension, all while keeping the model's overall parameter footprint small.

This is particularly powerful when adapting massively multilingual models that carry a lot of "dead weight" in their vocabularies. For instance, C. M. Downey et al. replaced a massive 250k-token generic multilingual vocabulary with a specialized 16k-token vocabulary tailored to the Uralic language family. This focused the model's representational capacity, shortened sequence lengths, and eliminated computational noise from unrelated languages.

To avoid increasing vocabulary size at all, Yuta Nozaki et al. developed a strategy called Vocabulary Replacement Continued Pretraining (VRCP) in VRCP: Vocabulary Replacement Continued Pretraining for Efficient Multilingual Language Models. VRCP identifies low-frequency source tokens and directly replaces them with high-frequency target language tokens, improving token efficiency while maintaining a perfectly stable GPU memory footprint.

Why It Works: The Mechanics of Efficiency

Altering the vocabulary yields dramatic performance and speed improvements due to two core computational mechanics:

  1. Drastic Sequence Reduction: By compressing text into fewer tokens, the model performs fewer autoregressive generation steps. Atsuki Yamaguchi et al. demonstrate in Adapting Chat Language Models Using Only Target Unlabeled Language Data that vocabulary expansion can mitigate token fragmentation to deliver inference speedups of 2x to 6x for low-resource languages. Furthermore, because a single context window can now hold more words, the effective context length of the model is implicitly multiplied (M Saiful Bari et al.; Shahriar Kabir Nahin et al.).
  2. Parameter and Embedding Specialization: In a standard multilingual model, a massive portion of the parameters is tied up in the embedding layers to support thousands of languages simultaneously. When specializing a model for a single language or family, keeping these unused tokens introduces unnecessary overhead. Pranaydeep Singh et al. and Seungyoon Lee et al. show that isolating and reducing the vocabulary to target-only tokens shrinks the overall model size—sometimes by more than 50% (Konstantin Dobler et al.)—which accelerates training and inference while preserving or improving downstream reasoning capabilities.

Aligning the Semantic Space

The central challenge of vocabulary adaptation is initialization. When you introduce thousands of new tokens or swap a tokenizer entirely, the model's existing transformer layers have no idea what these new token embeddings mean. If initialized randomly, the model requires massive, expensive pre-training datasets to relearn basic semantics.

To bypass this bottleneck, recent work focuses on mathematically aligning new tokens with the pre-trained model's existing semantic space.

[New Tokenizer] ---> Find Overlapping Subwords ---> [Sparsemax Weights] ---> [Original Embeddings]
                                                                                     |
                                                                                     v
                                                                        [Initialized New Embedding]

In Effective Embedding Initialization for Monolingual Specialization of Multilingual Models, Konstantin Dobler et al. introduced FOCUS (Fast Overlapping Token Combinations Using Sparsemax). FOCUS initializes new language-specific embeddings by looking at how the new tokens would be broken down by the old tokenizer. It uses a sparsemax function to combine the embeddings of those overlapping original tokens, successfully preserving the original model's semantic knowledge and drastically accelerating training.

Alternatively, researchers can use smaller, language-specific "helper" models to guide the alignment. Giovanni Puccetti et al. developed Semantic Alignment Vocabulary Adaptation (SAVA) in Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation. SAVA maps the embedding space of a small Italian helper model onto a larger English model, reducing token fertility by up to 25% while safely pruning parameters from the base model.

Similarly, Seungyoon Lee et al. proposed SALT (Semantic Aware Linear Transfer), which uses a linear regression framework to recycle the rich semantic knowledge of small, language-specific pre-trained models. By projecting these specialized representations directly into the giant LLM's space, they achieve faster convergence and superior cross-lingual transfer.

By treating tokenization not as a fixed constraint but as an adaptable layer, these techniques effectively eliminate the "English tax," allowing large language models to operate efficiently in any language.

Key Foundational Work

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023) — contributed 2 concept elements to this direction.
When the Student Becomes the Master: Learning Better and Smaller Monolingual Models from mBERT (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_tokenization_efficiency_0716dd91,
  title = {tokenization efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/tokenization-efficiency/rdr_0716dd912cac4abaa061e21a6019eb95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_tokenization_efficiency_0716dd91_v1,
  title = {tokenization efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/tokenization-efficiency/rdr_0716dd912cac4abaa061e21a6019eb95/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake