TL;DR
Fine-tuning a large language model (LLM) doesn't erase its "inherited" traits—such as vocabulary bias, cultural alignment, and licensing constraints—which are baked into the base weights. This paper quantifies the "tokenization premium" that makes LLMs significantly more expensive and less capable for Global South languages and proposes a "sovereignty audit" to unmask the true origins of supposedly local models.
What Problem Does This Paper Address?
As nations and organizations race to announce "sovereign" LLMs, the reality is that most are fine-tuned versions of a few dominant model families (like Llama or Mistral). The author argues that fine-tuning is a superficial "coat of paint" that fails to change the fundamental properties of a model.
This creates two major issues for the Global South:
- The Tokenization Premium: Because tokenizers are often optimized for English, other languages require more tokens to represent the same text. This makes the models more expensive to run (per-token pricing) and effectively shrinks their context window for non-English users.
- The Sovereignty Illusion: A model labeled as "national" still carries the cultural biases, memorized data, and legal restrictions of its American or Chinese base model.
Methods
The paper introduces three primary contributions:
- Tokenization Measurement: The author calculates a "premium" ($P$)—the ratio of tokens required for a language compared to English for the same content. Crucially, the method corrects for "unknown-token collapse," where a tokenizer might appear efficient only because it is failing to recognize a script and mapping it to a single "unknown" character.
- Sovereignty Audit: An eight-test "black-box" procedure that allows users to probe a model's inheritance without needing access to its weights. These tests include checking for instruction leakage, memorization of specific training data, and consistency across sessions.
- A New Design Regime: The author argues for shifting optimization targets from raw accuracy to "accuracy per joule" and "calibration." This involves using metrics like Expected Calibration Error (ECE) that prevent a model from inflating its score by simply refusing to answer difficult questions (abstention).
Datasets
- UDHR in Unicode: The study uses 27 parallel articles of the Universal Declaration of Human Rights across 31 languages. This provides a legally reviewed, semantically identical text for fair price and efficiency comparisons.
- Internal Calibration Benchmark: Used for preliminary measurements of a proprietary architecture, consisting of a mix of in-domain and out-of-domain questions.
- Tokenizers Tested: Seven tokenizers including Pythia, Mistral, Qwen 2.5, DeepSeek-V3, OpenAI’s cl100k and o200k, and the Brazilian "Tucano."
Key Insights and Ideas
- Inheritance is Sticky: Fine-tuning mostly teaches format and style; it does not reliably add new knowledge. Facts introduced during fine-tuning are learned slowly and often increase hallucinations.
- The "Tucano" Effect: Surprisingly, a tokenizer built specifically for Portuguese (Tucano) outperformed all global tokenizers for 11 out of 13 Latin-script Global South languages, including Indigenous languages like Guarani and Quechua. This suggests that regional orthographic overlap is a powerful, underutilized tool for model efficiency.
- The Sovereignty Audit: The paper provides a practical framework for "buying" organizations to verify if a model is truly local.
Key Results
- Extreme Premiums: In older tokenizers, Amharic costs 10x more than English. Even in the state-of-the-art
o200ktokenizer, it remains 6.8x more expensive. - Indigenous Languages: Languages like Guarani, Quechua, and Aymara cost between 1.6x and 2.7x more than English under global tokenizers.
- Efficiency Gains: Using the Portuguese-built Tucano tokenizer reduced the premium for Guarani and Quechua by about 24% compared to the best global commercial tokenizer.
Limitations and Future Work
- Register Bias: The measurements rely on the UDHR, which is a formal legal text. Results might differ for everyday conversational language.
- Orthographic vs. Linguistic: It is unclear if the Tucano tokenizer's success is due to linguistic similarities or merely shared alphabetic patterns; the author suggests testing against a Spanish-built vocabulary to isolate this effect.
- Preliminary Data: The measurements in Section 6 are based on a small, internal sample from a proprietary architecture and have not been independently reproduced.
- Conflict of Interest: The author is the founder of LUA Vision, the company developing the reference architecture mentioned in the paper.
Summary updated