No article text was recorded for this version.
tokenizer fertility
Version 1 of 118 papers30 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
- Blog: MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
- Blog: IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
- Blog: MEGA: Multilingual Evaluation of Generative AI
- Blog: FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data
- Blog: Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
- Blog: Direct Speech Translation Toward High-Quality, Inclusive, and Augmented Systems
- Blog: RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
- Blog: Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
- Blog: Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
- Blog: Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection
- Blog: TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking
- Blog: Exploring Design Choices for Building Language-Specific LLMs
- Blog: On Measuring and Mitigating Biased Inferences of Word Embeddings
- Blog: The Token Tax: Systematic Bias in Multilingual Tokenization
- Blog: ALLaM: Large Language Models for Arabic and English
- Blog: Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough
- Blog: Tokenization as Cultural Erasure: How Corpus Composition Shapes the Representation of Aymara Morphology in NLP Systems
History & citation
Current · Permanent link
- This direction split from Mitigating the Multilingual Tokenization Tax in Language Models (78% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_tokenizer_fertility_ac39234c,
title = {tokenizer fertility},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/tokenizer-fertility/rdr_ac39234cd43840a79f7f627f3ff15546},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}This exact version (version 1, frozen text):
@misc{lacuna_tokenizer_fertility_ac39234c_v1,
title = {tokenizer fertility},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/tokenizer-fertility/rdr_ac39234cd43840a79f7f627f3ff15546/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}