Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South

Paulo CamaraarXiv 2026 · arXiv

TL;DR

Fine-tuning a large language model (LLM) doesn't erase its "inherited" traits—such as vocabulary bias, cultural alignment, and licensing constraints—which are baked into the base weights. This paper quantifies the "tokenization premium" that makes LLMs significantly more expensive and less capable for Global South languages and proposes a "sovereignty audit" to unmask the true origins of supposedly local models.

What Problem Does This Paper Address?

As nations and organizations race to announce "sovereign" LLMs, the reality is that most are fine-tuned versions of a few dominant model families (like Llama or Mistral). The author argues that fine-tuning is a superficial "coat of paint" that fails to change the fundamental properties of a model.

This creates two major issues for the Global South:

  1. The Tokenization Premium: Because tokenizers are often optimized for English, other languages require more tokens to represent the same text. This makes the models more expensive to run (per-token pricing) and effectively shrinks their context window for non-English users.
  2. The Sovereignty Illusion: A model labeled as "national" still carries the cultural biases, memorized data, and legal restrictions of its American or Chinese base model.

Methods

The paper introduces three primary contributions:

  • Tokenization Measurement: The author calculates a "premium" ($P$)—the ratio of tokens required for a language compared to English for the same content. Crucially, the method corrects for "unknown-token collapse," where a tokenizer might appear efficient only because it is failing to recognize a script and mapping it to a single "unknown" character.
  • Sovereignty Audit: An eight-test "black-box" procedure that allows users to probe a model's inheritance without needing access to its weights. These tests include checking for instruction leakage, memorization of specific training data, and consistency across sessions.
  • A New Design Regime: The author argues for shifting optimization targets from raw accuracy to "accuracy per joule" and "calibration." This involves using metrics like Expected Calibration Error (ECE) that prevent a model from inflating its score by simply refusing to answer difficult questions (abstention).

Datasets

  • UDHR in Unicode: The study uses 27 parallel articles of the Universal Declaration of Human Rights across 31 languages. This provides a legally reviewed, semantically identical text for fair price and efficiency comparisons.
  • Internal Calibration Benchmark: Used for preliminary measurements of a proprietary architecture, consisting of a mix of in-domain and out-of-domain questions.
  • Tokenizers Tested: Seven tokenizers including Pythia, Mistral, Qwen 2.5, DeepSeek-V3, OpenAI’s cl100k and o200k, and the Brazilian "Tucano."

Key Insights and Ideas

  • Inheritance is Sticky: Fine-tuning mostly teaches format and style; it does not reliably add new knowledge. Facts introduced during fine-tuning are learned slowly and often increase hallucinations.
  • The "Tucano" Effect: Surprisingly, a tokenizer built specifically for Portuguese (Tucano) outperformed all global tokenizers for 11 out of 13 Latin-script Global South languages, including Indigenous languages like Guarani and Quechua. This suggests that regional orthographic overlap is a powerful, underutilized tool for model efficiency.
  • The Sovereignty Audit: The paper provides a practical framework for "buying" organizations to verify if a model is truly local.
The eight-test black-box sovereignty audit proposed by the author.

Key Results

  • Extreme Premiums: In older tokenizers, Amharic costs 10x more than English. Even in the state-of-the-art o200k tokenizer, it remains 6.8x more expensive.
  • Indigenous Languages: Languages like Guarani, Quechua, and Aymara cost between 1.6x and 2.7x more than English under global tokenizers.
  • Efficiency Gains: Using the Portuguese-built Tucano tokenizer reduced the premium for Guarani and Quechua by about 24% compared to the best global commercial tokenizer.
Detailed tokenization premiums across various languages and model families. Bold values indicate where the regional Tucano tokenizer outperforms global models.

Limitations and Future Work

  • Register Bias: The measurements rely on the UDHR, which is a formal legal text. Results might differ for everyday conversational language.
  • Orthographic vs. Linguistic: It is unclear if the Tucano tokenizer's success is due to linguistic similarities or merely shared alphabetic patterns; the author suggests testing against a Spanish-built vocabulary to isolate this effect.
  • Preliminary Data: The measurements in Section 6 are based on a small, internal sample from a proprietary architecture and have not been independently reproduced.
  • Conflict of Interest: The author is the founder of LUA Vision, the company developing the reference architecture mentioned in the paper.

Summary updated

Abstract

Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families. We argue that fine-tuning leaves the decisive properties of a model inherited from its base: vocabulary, knowledge, cultural alignment, memorized data and license. We contribute (i) an updated, reproducible measurement of the tokenization premium across 31 languages and 7 tokenizers released in 2023-2024, on 27 articles of the Universal Declaration of Human Rights, with explicit correction for unknown-token collapse. Premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one; Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. A vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua. (ii) An eight-test black-box sovereignty audit that a buyer can run without access to weights. (iii) A regime argument: under energy and liability constraints, accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives, with metrics that abstention cannot inflate. We close with preliminary internal measurements from a proprietary architecture built around these objectives, reported with their limitations. Data and measurement code are included.
BibTeX
@misc{camara2026inherited,
  author = {Paulo Camara},
  title = {{Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South}},
  year = {2026},
  eprint = {2611.13441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13441},
}
Figure