Most modern language models do not process text as words or characters, but as subwords. Algorithms like Byte-Pair Encoding (BPE) or WordPiece were designed to solve the "out-of-vocabulary" problem by breaking rare words into frequent fragments. However, these algorithms prioritize statistical frequency and data compression over linguistic structure. This creates a fundamental rift: the units the machine learns from often bear no relation to the units of meaning (morphemes) used by humans.
The Statistical-Linguistic Mismatch
Subword tokenization is typically language-agnostic, relying on substring frequency to determine where to split a word. As Peters and Martins observe, this creates a gap where machine-generated tokens fail to align with meaningful morphemes. For example, a tokenizer might split "unhappy" into "unh" and "appy" rather than the linguistically correct "un-" and "happy."
This misalignment is particularly damaging for morphologically rich languages (MRLs). In languages like Basque, Agerri et al. et al. demonstrate that standard tokenizers produce linguistically nonsensical fragments, preventing models from capturing the relationship between roots and functional affixes. When a tokenizer breaks a word into fragments that lack semantic or grammatical meaning, the model's ability to represent the language's underlying rules is significantly diminished.
The Problem of High Fertility
When a model encounters a language it wasn't primarily trained on—or a language with complex word structures—it often resorts to over-segmentation. This is measured by "fertility," or the number of tokens produced per word.
Mekonnen et al. show that high subword fertility in Amharic breaks words into meaningless fragments, which directly degrades the performance of dense retrieval systems. Similarly, Patil argues that vocabularies biased toward high-resource languages force low-resource words into semantically opaque sub-units. This excessive fragmentation increases sequence length and makes it harder for the model to capture coherent semantic meaning, as the signal is diluted across too many arbitrary steps.
Agglutination and Non-Concatenative Morphology
The limitations of subwords are most visible in specific linguistic structures:
- Agglutinative Languages: In languages like Turkish or Tibetan, words are formed by stringing together multiple morphemes. Verma et al. note that standard tokenizers struggle to capture these patterns, leading to lower accuracy. Lv et al. further highlight that for scripts with vertical character combinations, tokenizers often break characters at inappropriate points, disrupting the model's understanding of syllabic boundaries.
- Non-Concatenative Morphology: Some languages, like Arabic or Ethiopic scripts, use root-and-pattern systems where meaning is modified by changing internal vowels rather than adding prefixes or suffixes. Heinzerling and Strube argue that BPE-style methods are fundamentally ill-equipped for these systems, as they assume meaning is built by concatenating substrings.
Artificial Perplexity and Grammatical Errors
Tokenization isn't just a preprocessing step; it changes how the model perceives the "difficulty" of a sentence. Gambardella et al. found that in Japanese, adding a grammatical suffix can trigger "emergency" byte-level processing. This causes an artificial spike in perplexity, leading the model to sometimes prefer ungrammatical sentences simply because they are easier to tokenize.
Furthermore, Truong et al. show that while tokenizers often "break" critical markers like negative affixes (e.g., un- or -less), larger models must work harder to overcome these misleading tokens by leveraging internal word knowledge. This suggests that subword tokenization acts as a "bottleneck" that models must learn to ignore or circumvent.
Toward Morphology-Aware Architectures
To fix these issues, researchers have proposed several alternatives to standard statistical subwords:
- Linguistically-Informed Segmentation: Huck et al. demonstrate that integrating explicit compound and suffix splitting into the tokenization process significantly boosts translation quality for German.
- Tokenization-Free Models: Clark et al. proposed CANINE, which operates directly on Unicode characters, bypassing subwords entirely to remain robust against typos and morphological complexity.
- Hybrid Hierarchical Models: Ataman et al. suggest hierarchical decoders that respect word boundaries while processing characters, combining the flexibility of character-level models with the efficiency of word-level ones.
- Specialized Monolingual Models: Nzeyimana and Rubungo developed KinyaBERT, which replaces BPE with a transformer that explicitly encodes morphological parses (stems and affixes), outperforming standard BERT with fewer parameters.
While Arnett et al. suggest that some performance gaps in MRLs disappear when accounting for the "byte premium" (the storage cost of information), the consensus across the field remains that aligning machine units with linguistic reality is essential for robust, cross-lingual understanding.