The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India

Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin SamarXiv 2026 · arXiv

This paper, titled "The Indic Token Tax in 2026: Measuring Tokenizer Disparity and Its Cost for the Languages of India," provides a rigorous audit of how modern Large Language Model (LLM) tokenizers treat the diverse languages of the Indian subcontinent. It quantifies a persistent "token tax"—the extra cost and reduced efficiency faced by non-English users—and evaluates how well the latest 2024–2026 models are closing this gap.

TL;DR

The "token tax" remains a significant barrier for Indian languages in 2026; for example, Marathi text can cost 82% more than English on GPT-4o due to inefficient tokenization. While Indic-focused models like Sarvam-30B significantly improve parity, speakers of languages with unique scripts like Odia and Santali still face costs up to 13.7x higher than English.

What Problem Does This Paper Address?

Commercial LLM APIs typically bill by the token. Because most tokenizers are trained on English-centric corpora, they tend to fragment non-English text into many small, inefficient pieces (often down to individual bytes).

This creates a triple-threat "tax" for global users:

  1. Financial: You pay more for the same amount of information.
  2. Latency: Models generate text one token at a time; more tokens mean slower responses.
  3. Context: Model "memory" (context windows) is measured in tokens; inefficient tokenization effectively shrinks the amount of text a model can "see" at once.

The authors set out to measure this disparity across nine major public tokenizers and 25 Indian languages, including the increasingly common "Hinglish" (Hindi written in Latin script).

Methods

The researchers calculated a parity ratio: the number of tokens required for a language $L$ divided by the tokens required for the same meaning in English. A ratio of 1.0 represents perfect parity.

They evaluated nine tokenizers:

  • OpenAI: GPT-4 (cl100k_base) and GPT-4o (o200k_base).
  • Open Weights: Qwen2.5, DeepSeek-V3, and Mistral-Nemo.
  • Indic-Specific: Four models from Sarvam AI (Sarvam-1, Sarvam-M, Sarvam-30B, and Sarvam-105B).

Beyond raw counts, they conducted a rupee cost analysis using 2026 list prices and measured morpheme boundary alignment to see if tokenizers were splitting words in linguistically meaningful ways or just arbitrary chunks.

Datasets

  • FLORES-200: 2,009 professionally translated sentences across 25 Indian language/script pairs.
  • PHINC: 13,732 pairs of Hinglish-English social media posts to measure "romanized" text.
  • Dakshina: 10,000 Wikipedia strings per language, provided in both native script and romanized forms.
  • MorphTok: Gold-standard morpheme segmentations for Hindi and Marathi used to test tokenizer "linguistic intelligence."

Key Insights and Ideas

  • The "Romanization Reversal": On older, English-heavy tokenizers (like GPT-4), typing in Latin letters (romanization) was a "hack" to save money, cutting costs by up to 75%. However, for new Indic-focused tokenizers like Sarvam-1, this effect is reversed: romanized text actually costs more than native script because the tokenizer is optimized for Devanagari.
  • Script Marginalization: Even "multilingual" models have massive blind spots. While Hindi parity is improving, languages with distinct scripts like Odia (5.01x tax on GPT-4o) and Santali (13.72x tax) are being left behind.
  • Linguistic Blindness: None of the tokenizers—even the Indic-focused ones—align well with morpheme boundaries (the smallest units of meaning in a word). Most tokenizers still split words in ways that make little linguistic sense, which may hinder the model's ability to generalize.
  • Sarvam's Success: The Sarvam-30B/105B models are the most "fair," staying below a 2.6x tax across all 25 languages, suggesting that targeted training on regional data is the most effective path to parity.
Fig. 1. Parity ratio versus English for each tokenizer, with 95% bootstrap confidence intervals. The dashed line marks parity with English.

Key Results

  • GPT-4o Improvements: GPT-4o's o200k_base tokenizer significantly reduced the "tax" compared to GPT-4, bringing Hindi down from 4.78x to 1.57x.
  • The Cost of Business: In a hypothetical scenario of a support bot handling 100,000 chats a month, using GPT-4o would cost INR 56,640 in English but INR 102,829 in Marathi—a "tax" of over 46,000 rupees per month for the exact same service.
  • The Heatmap of Inequality: As shown in Figure 2, while most languages cluster between 1.5x and 3.0x on modern models, the "red zone" for Odia and Santali remains extreme across almost all non-Indic tokenizers.
Fig. 2. Parity ratio versus English on FLORES-200 for every tokenizer (rows, sorted by median) and language of India (columns, sorted by median), on a log color scale.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Overlap: They did not check if the FLORES-200 test sentences were present in the models' training data (data leakage).
  • Formal vs. Casual: FLORES-200 is formal, Wikipedia-style text. Real-world conversational text might tokenize differently.
  • Missing Languages: Some official Indian languages (Bodo, Dogri, Konkani) were missing from the primary dataset.
  • Future Directions: The authors suggest future work should include evaluating closed-source models like Claude (via its token-counting API) and testing on more conversational benchmarks like IN22-Conv.

Summary updated

Abstract

Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at https://github.com/HavonAI/indic-token-tax.
BibTeX
@misc{arimbur2026indic,
  author = {Johin Johny Arimbur and Ankur Ashit Ghosh and Shervin Sam},
  title = {{The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India}},
  year = {2026},
  eprint = {2611.13058},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13058},
}
Figure