No article text was recorded for this version.
cross-lingual tokenization inefficiency
Version 1 of 134 papers42 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Language Model Tokenizers Introduce Unfairness Between Languages
- Blog: Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
- Blog: An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
- Blog: MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages
- Blog: VRCP: Vocabulary Replacement Continued Pretraining for Efficient Multilingual Language Models
- Blog: A Principled Framework for Evaluating on Typologically Diverse Languages
- Blog: Retrofitting Large Language Models with Dynamic Tokenization
- Blog: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
- Blog: Zero-Shot Tokenizer Transfer
- Blog: Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation
- Blog: Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages
- Blog: Adapting Chat Language Models Using Only Target Unlabeled Language Data
- Blog: From Bias to Balance: How Multilingual Dataset Composition Affects Tokenizer Performance Across Languages
- Blog: Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages
- Blog: Efficient Continual Pre-training of LLMs for Low-resource Languages
- Blog: Automatic Short Answer Grading for Finnish with ChatGPT
- Blog: Understanding and Mitigating Language Confusion in LLMs
- Blog: See the Text: From Tokenization to Visual Reading
- Blog: Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages
- Blog: Measuring Cross-lingual Transfer in Bytes
- Blog: SeaLLMs - Large Language Models for Southeast Asia
- Blog: Tokenizer Choice For LLM Training: Negligible or Crucial?
- Blog: Small Languages, Big Models: A Study of Continual Training on Languages of Norway
- Blog: BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
- Blog: Tuning Multilingual Transformers for Language-Specific Named Entity Recognition
- Blog: A Formal Perspective on Byte-Pair Encoding
- Blog: Learning to Route Languages for Multilingual Policy Optimization
- Blog: GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages
- Blog: MonoByte: A Pool of Monolingual Byte-level Language Models
- Blog: Large-Scale Contextualised Language Modelling for Norwegian
- Blog: Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives
- Blog: Provably Secure Disambiguating Neural Linguistic Steganography
- Blog: Multi-word Tokenization for Sequence Compression
- Blog: On the Challenges and Opportunities in Generative AI
History & citation
Current · Permanent link
- This direction split from Mitigating the Multilingual Tokenization Tax in Language Models (74% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_cross_lingual_tokenization_inefficiency_440aeea7,
title = {cross-lingual tokenization inefficiency},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/cross-lingual-tokenization-inefficiency/rdr_440aeea779d242dcb020e933936b94b8},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_cross_lingual_tokenization_inefficiency_440aeea7_v1,
title = {cross-lingual tokenization inefficiency},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/cross-lingual-tokenization-inefficiency/rdr_440aeea779d242dcb020e933936b94b8/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}