What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and Hausa

Abdul Rehman Antall, Baidya Nath SahaarXiv 2026 · arXiv

TL;DR

When researchers claim a "morphology-aware" tokenizer outperforms standard Byte-Pair Encoding (BPE), they are often conflating three distinct factors: token size (granularity), linguistic grouping (root assignment), and how sub-tokens are aggregated for tasks (readout). This paper disentangles these effects, proving that while linguistic roots do provide a genuine performance boost in Amharic NER, the choice of "readout" strategy is a massive hidden confounder that can completely reverse the results of a tokenizer comparison.

What Problem Does This Paper Address?

In Natural Language Processing, tokenization is the first and often most overlooked decision. For morphologically rich languages like Amharic (which uses a non-concatenative root-and-pattern system), standard statistical tokenizers like BPE often fail to capture the underlying linguistic structure.

Researchers frequently propose new, linguistically-informed tokenizers and justify them by showing a gain over BPE. However, the authors argue that these comparisons are often "apples-to-oranges." A morphology-aware tokenizer doesn't just change what the units are; it usually changes how many units there are (granularity) and how the model must read them to make a prediction (readout). Without isolating these variables, we don't actually know if the "linguistic" part of the tokenizer is doing the heavy lifting or if we're just seeing the side effects of finer segmentation or a lucky implementation choice.

Methods

The authors introduce a rigorous methodology to isolate these effects using Amharic (Ge’ez script) and Hausa (Latin script) as case studies.

  1. Decomposition: They split the total performance difference between a morphology-aware tokenizer ($T$) and BPE ($B$) into a granularity component ($\Delta_{gran}$) and a root assignment component ($\Delta_{root}$).
  2. Controlled Tokenizers: To isolate $\Delta_{root}$, they created a ROOTPAT tokenizer that explicitly separates Amharic roots from their vocalic templates. They then compared this to:
  3. Scrambled Roots: A control that keeps the exact same segmentation and fertility but replaces real linguistic roots with random identifiers.
  4. Permuted Roots: A control that preserves the frequency distribution of roots but shuffles which words they are assigned to.
  5. Readout Sweep: They tested two ways of reading word labels from sub-tokens: using only the first sub-token (the common default) versus mean pooling across all sub-tokens of a word.
  6. Replication Ledger: To ensure robustness, they ran the entire experiment twice (Study I and Study II) with independent analysers and codebases, reporting only what held true in both.
Figure 1: The experimental design for disentangling granularity, root assignment, and readout.

Datasets

  • NER: MasakhaNER 1.0 for Amharic and Hausa (Named Entity Recognition).
  • Translation: Filtered OPUS-100 data for English to Amharic/Hausa.
  • Morphology: UniMorph (Amharic paradigms) and HornMorpho (root analyser).

Key Insights and Ideas

  • The Readout Confound is Massive: In models trained from scratch, the choice of readout (first-token vs. mean-pooling) can swing F1 scores by over 13 points. Crucially, using the "first-token" default often makes morphology-aware tokenizers look worse than BPE, while mean-pooling makes them look better. This implementation detail can entirely flip the conclusion of a paper.
  • Linguistic Roots Provide a Real Boost: Even after controlling for granularity and frequency statistics, assigning the "real" linguistic root to Amharic words improved NER by ~4.7 F1 points over the scrambled control. This confirms that grouping related word forms (e.g., "he broke" and "breaker") helps the model generalize, especially in low-resource settings.
  • Granularity is Language-Dependent: Finer segmentation (more tokens per word) helped Amharic NER but significantly hurt Hausa. This is likely because Amharic's Ge'ez script fuses consonants and vowels into single glyphs; breaking these down helps the model see the "sub-glyph" patterns. In Hausa, which uses a Latin script, the default units are already close to the optimal level of detail.

Key Results

  • Root Assignment: Real roots beat scrambled roots by +4.86 F1 (Study I) and +4.65 F1 (Study II) in Amharic NER.
  • Readout Reversal: In Study I, switching from first-token readout to mean pooling changed the comparison of the "analyser" tokenizer vs. BPE from -2.26 F1 (a loss) to +9.11 F1 (a major win).
  • Pretrained Models: The readout effect nearly vanishes in large pretrained encoders like XLM-R (<1 F1 difference), likely because their high degree of contextualization makes the specific pooling strategy less critical.
Figure 2: Decomposition of the Amharic NER comparison with BPE (pooled readout, five seeds; in Study I, “real” is the analyser arm). The first three rows are the same contrasts in both studies: the root component (blue) reproduces, whereas the granularity component (green) changes sign and takes the total (black) with it. Rows below the dashed line are further controls of each study. The far-right seed in (b) is one high-scoring real run.

Limitations and Future Work

  • Model Scale: The most dramatic effects were found in small Transformers (1M-2M parameters) trained from scratch. While the authors tested pretrained encoders, the full "scrambled root" controls cannot be easily run behind a fixed, pretrained tokenizer.
  • Task Variety: The findings are strongest for token-level classification (NER). Results for generative tasks like Machine Translation were less conclusive due to lower coverage of the root analyser on the MT dataset.
  • Language Scope: The study focuses on two Afroasiatic languages. Expanding this to other non-concatenative languages (like Hebrew or Arabic) or different script types would further validate the "orthographic explanation" for granularity.

The authors conclude with a strong recommendation: Tokenizer papers must report their readout strategy and use scrambled controls to prove that their gains actually come from linguistic structure rather than just accidental implementation details.

Summary updated

Abstract

Morphology-aware tokenizers are expected to help morphologically rich languages, and their reported gains over byte-pair encoding (BPE) are usually credited to their linguistic units. A comparison with BPE cannot support this attribution, since such a tokenizer also segments text more finely (granularity) and, on token-level tasks, changes how much the sub-token readout matters. We separate both factors from the effect of grouping words by their roots (root assignment) in Amharic, a Semitic language whose Ge'ez abugida hides consonantal roots inside syllabic glyphs, with Latin-script Hausa as an orthographic contrast. Our design combines a lossless root-and-pattern tokenizer with controls that change one factor at a time, in two independent implementations. In both, real root assignment improves Amharic named-entity recognition over scrambled roots (+4.9 and +4.7 F1), and root-frequency structure explains little of this gain. In small from-scratch models, the readout choice moves F1 by up to 13.4 points and reverses the sign of every comparison of the root-and-pattern arms with BPE, whereas in two pretrained encoders it moves F1 by less than one point. The granularity effect changes sign across languages and implementations. We recommend decomposing tokenizer effects, and reporting the readout, before a gain is attributed to linguistic structure.
BibTeX
@misc{antall2026what,
  author = {Abdul Rehman Antall and Baidya Nath Saha},
  title = {{What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and Hausa}},
  year = {2026},
  eprint = {2611.14818},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14818},
}
Figure