The Token Tax: Systematic Bias in Multilingual Tokenization

Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, David Adelani, Cody Carroll

Abstract

Tokenization serves as the foundational interface between human language and Large Language Models (LLMs). However, for many non-English languages, this interface acts as a "Token Tax." This paper investigates how inefficient tokenization—measured by "fertility"—negatively impacts both model performance and computational costs. Analyzing 10 prominent LLMs across 16 African languages using the AfriMMLU benchmark, we find that higher fertility consistently predicts lower accuracy and leads to a nonlinear explosion in training and inference expenses. While reasoning-heavy models mitigate some performance gaps, the structural inequality of tokenization remains a significant barrier to global AI equity.

Introduction

In the current landscape of Large Language Models (LLMs), tokenization is often treated as a standard preprocessing step. However, for the majority of the world's languages, tokenization is far from neutral. Most state-of-the-art models utilize tokenizers optimized for English, which results in a "Token Tax" for non-English speakers. This tax manifests as a systematic bias where morphologically complex or low-resource languages are broken into numerous small, often semantically vacuous fragments. This research quantifies the impact of this inefficiency, demonstrating that it is a primary driver of performance degradation and economic exclusion in AI.

The Mechanics of the Token Tax

The central metric for measuring tokenization efficiency is fertility: the ratio of tokens produced per word. A high fertility rate indicates that a tokenizer is struggling to represent a language efficiently. This creates a two-pronged "tax":

  1. The Performance Tax: Transformers operate within a finite context window. When a language has high fertility, a single sentence consumes more of that window. Furthermore, the model must learn to reconstruct meaning from fragmented tokens, increasing the cognitive load on the attention mechanism. This leads to a direct drop in accuracy.
  2. The Economic Tax: The computational complexity of the self-attention mechanism scales quadratically ($O(n^2)$) with sequence length. Consequently, a language that requires twice as many tokens as English to convey the same meaning is not merely twice as expensive—it is approximately four times as expensive to process in terms of compute, energy, and time.

Methodology

To investigate these effects, we conducted a systematic evaluation across 10 frontier LLMs, including Llama 3.1, GPT-4o, Claude 3.5, and specialized reasoning models such as DeepSeek-R1 and OpenAI o1.

The study utilized the AfriMMLU benchmark, which comprises 9,000 Multiple Choice Question Answering (MCQA) items across 16 African languages (including Swahili, Yoruba, Amharic, and Hausa). These items span five diverse subjects: Elementary Mathematics, Global Facts, Geography, Macroeconomics, and International Law. By comparing performance against English and French baselines, we utilized linear regressions and mixed-effects models to isolate the impact of fertility on model accuracy.

Key Findings

Fertility as a Performance Predictor

Our analysis reveals that fertility is a remarkably reliable predictor of model failure. Across all tested models and subjects, higher fertility explained between 20% and 50% of the variance in accuracy. Specifically, for every additional token required per word, model accuracy dropped by an average of 8 to 18 percentage points. This suggests that the "multilingual gap" in AI is, in large part, a structural artifact of how we represent text.

The Reasoning Buffer

A significant discovery in this research is the role of "reasoning" models (e.g., DeepSeek-R1 and OpenAI o1). While standard models showed a performance gap of approximately 25 points between English and African languages, reasoning models narrowed this gap to 12–14 points. This suggests that enhanced internal logic and "chain-of-thought" capabilities can partially compensate for poor data representation, though they do not eliminate the underlying inefficiency.

The Economic and Temporal Burden

The "Economic Tax" presents a formidable barrier to AI adoption in the Global South. Our modeling shows that training a model like Llama 3.1 405B on English-heavy data costs approximately $105 million. However, for a language with double the fertility of English, the same training run would cost $420 million and take four times as long to complete.

This disparity extends to inference. Generating a response in a high-fertility language results in significantly higher latency and higher costs for users of API-based models that charge per token. This effectively prices out linguistic communities that are already underrepresented in the digital economy.

Conclusion and Future Directions

The Token Tax is not an inevitable feature of AI; it is a design choice. Current tokenization strategies prioritize English at the expense of global linguistic diversity. To move toward equitable AI, the research community must prioritize:

  • Morphologically Aware Tokenization: Developing tokenizers that respect the linguistic structure of diverse language families.
  • Fair Pricing Models: Moving away from per-token pricing, which inherently penalizes non-English speakers, toward "meaning-based" or "character-based" billing.
  • Architectural Innovation: Exploring efficient attention mechanisms (such as linear attention) that can mitigate the quadratic cost penalty associated with long token sequences.

Addressing the Token Tax is essential for ensuring that the benefits of the AI revolution are accessible to all, regardless of the language they speak.

Summary updated

BibTeX
@inproceedings{lundin2026token,
  author = {Jessica M. Lundin and Ada Zhang and Nihal Karim and Hamza Louzan and Victor Wei and David Adelani and Cody Carroll},
  title = {{The Token Tax: Systematic Bias in Multilingual Tokenization}},
  booktitle = {AfricaNLP @ EACL},
  year = {2026},
  url = {https://openreview.net/forum?id=zMOuykBYE2},
}
Figure