TL;DR
UTF-8 is the "invisible" standard for how language models read text, but this paper shows it’s far from optimal for every language. By testing 16 different encoding strategies, the authors reveal that "information-dense" scripts like Chinese and Korean see massive performance gains from more compressed encodings, while "information-sparse" scripts like Tamil and Georgian are surprisingly indifferent to how they are encoded.
What Problem Does This Paper Address?
Most modern Large Language Models (LLMs) process text as a sequence of bytes. Because UTF-8 is the universal standard for digital text, it has become the default "pre-tokenizer" for almost all models. However, UTF-8 is variable-length: a Latin character takes 1 byte, while a Chinese character might take 3 or 4.
This creates a "byte premium" for non-Latin scripts, leading to longer sequences and potential inequities in how models learn different languages. While researchers have proposed alternative encodings to improve fairness, we haven't had a systematic study of how these choices actually impact model performance. This paper asks: Does the way we turn characters into bytes actually change how well a model learns?
Methods
The authors conducted a massive sweep of 38 language-encoding pairs using a GPT-2-sized architecture. To make the comparison fair, they developed two key tools:
- AETHER Tokenizers: A hybrid framework that uses a specialized encoding for a target script (like Korean) but "falls back" to standard UTF-8 for everything else (like emojis or English loanwords). This ensures the model can still read any Unicode character.
- CuTF (Customized UTF): A novel, fixed-length encoding scheme designed to map a specific Unicode block (like Hangul Syllables) into the smallest possible vocabulary for a given sequence length.
They compared 16 different encoding methods, ranging from standard Unicode (UTF-8, UTF-16) to language-specific decompositions like Jamo for Korean and IDS for Chinese.
Datasets
The models were trained on high-quality monolingual corpora:
- Chinese: FineWeb-Edu-Chinese-v2
- Korean: AI Hub pretraining corpus
- Tamil: Sangraha dataset
- Georgian: FineWeb2
For evaluation, they used the FLORES dataset and a specialized metric called Macro-NLL. Unlike standard loss, Macro-NLL weights every unique character equally, preventing the results from being skewed by a few very common characters.
Key Insights and Ideas
- Orthographic Granularity is King: The "aha moment" of the paper is that encoding only matters if the characters themselves are "dense." In Chinese, one character is often a whole word (morpheme). In Korean, one character is a full syllable. For these scripts, compressing the encoding (fewer bytes per character) significantly boosts performance.
- The "Sparse" Exception: In Tamil and Georgian, characters represent individual consonants or vowels—units that carry very little information on their own. For these languages, the model is remarkably robust; it performs roughly the same whether a character is represented by 1 byte or 8 bytes.
- Fixed-Length > Variable-Length: For Korean, the best-performing model used Jamo (a fixed-length decomposition). When they tried a variable-length version of the same units, performance dropped. This suggests that models benefit from the predictable patterns of fixed-length encodings.
Key Results
The results show a stark divide between language types. In Chinese and Korean, there is a strong correlation between compression and performance (lower Macro-NLL).
- Chinese: A very strong correlation ($r = 0.911, p < 0.001$) between training steps per epoch (a proxy for compression) and performance.
- Korean: A strong correlation ($r = 0.886$) when excluding the unique Jamo encoding.
- Tamil & Georgian: No significant correlation. Even when encodings differed in length by a factor of 8x, the models performed similarly.
Limitations and Future Work
The authors acknowledge a few key hurdles for future research:
- Model Scale: These experiments were done on 120M parameter models. It remains to be seen if these encoding effects persist or diminish as models scale to billions of parameters.
- Monolingual Focus: The study focused on single-language models. In a massive multilingual model, different encodings might interact in complex ways.
- Generation Safety: Custom encodings can sometimes lead a model to generate "malformed" byte sequences that don't map back to valid text—a problem the authors left for future work.
Ultimately, the paper argues that text encoding should be a deliberate design choice tailored to a language's script, rather than a one-size-fits-all UTF-8 default.
Summary updated