FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages

Gebreslassie Teklu Reda, Yuval PinterarXiv 2026 · arXiv

TL;DR

FIDEL (FIdäl DEcomposition and Linearization) is a novel pre-tokenization framework designed for Ethiopic languages like Amharic. By "unfusing" the consonant-vowel combinations inherent in the abugida script, FIDEL exposes underlying linguistic roots, significantly improving model performance on tasks like news classification and named-entity recognition.

What Problem Does This Paper Address?

Most modern Large Language Models (LLMs) rely on subword tokenization (like BPE or UnigramLM). These algorithms assume that languages are "concatenative"—meaning you build words by sticking prefixes, roots, and suffixes together in a line.

However, Semitic languages (like Amharic, Tigrinya, and Ge’ez) use root-and-pattern morphology. Here, a core consonantal root (e.g., s-b-r for "break") is interleaved with vowels to create different meanings. The challenge is doubled for Ethiopic languages because they use an abugida script called fidäl. In this script, a single character represents a consonant-vowel pair (e.g., "በ" is bä). Because the consonant and vowel are fused into one Unicode codepoint, the underlying "skeleton" of the word is invisible to standard tokenizers. If a model can't see that two different-looking characters share the same root consonant, it struggles to generalize.

Methods

The authors introduce FIDEL, a reversible pre-tokenization scheme that operates in two main stages:

  1. Decomposition: It breaks each fidäl character into its base consonant and a separate vowel marker. They tested two layouts:
  2. Interleaved (CVCV): Vowel markers follow their consonants immediately.
  3. Grouped (CCCVVV): All consonants are listed first, followed by a sequence of vowel markers with position indices.
  4. Reduction (SPLINTER): Once the consonants are exposed, the authors adapt the SPLINTER algorithm. This iteratively prunes "templatic" material (affixes and specific vowel patterns) to reach the most frequent core root. For example, the word for "they shattered" (säbabbäru) is reduced to the root for "he broke" (säbbärä), with the changes recorded as special "reduction markers."

Crucially, this process is fully reversible. The model processes the "unzipped" text, but the output can be deterministically reconstructed back into standard fidäl script.

Datasets

The researchers evaluated FIDEL across three languages using a 385 MB Ge’ez-script corpus for pre-training. Evaluation was conducted on several established benchmarks:

  • NER: EthioNER and MasakhaNER.
  • Sentiment Analysis: AfriSenti.
  • News Classification: MasakhaNEWS.
  • POS Tagging: UD_Amharic-ATT.
  • Question Answering: AmQA (Extractive QA).

Key Insights and Ideas

  • The "Unzipping" Necessity: You cannot apply root-reduction to Ethiopic text without first decomposing the characters. Standard reduction on fused characters fails because the root consonants aren't contiguous substrings.
  • Task-Specific Utility: FIDEL is not a "silver bullet" for every task. It excels at stem-identity tasks (News Classification, NER) where knowing the core subject is vital. However, it can struggle with sentiment analysis, likely because the reduction process "compresses" inflectional cues (like negation markers) into opaque markers that are harder for the model to interpret.
  • Vocabulary Capacity Matters: The benefits of FIDEL are most pronounced at larger vocabulary sizes (e.g., 64k). At smaller sizes (16k), the overhead of storing extra vowel and reduction markers can "crowd out" the vocabulary, negating the structural gains.
  • Intrinsic Metrics are Deceptive: The paper highlights that common metrics used to judge tokenizers (like "tokens per word" or "fertility") do not reliably predict how well a model will perform on actual downstream NLP tasks.

Key Results

  • News Classification: FIDEL schemes consistently outperformed the baseline by 1 to 2 points across different vocabulary sizes.
  • Named-Entity Recognition (NER): In Amharic, the splinter_only scheme reached an F1 of 65.80 compared to the baseline's 62.79 (at 16k vocab).
  • Question Answering: Performance was preserved or slightly improved, with syllabic_then_spl achieving the highest F1 score (61.20 vs 60.55 baseline).
  • Efficiency: The interleaved decomposition (syllabic_then_splinter) proved to be the most token-efficient at high vocabulary counts, reducing the average number of tokens per word.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Language Imbalance: The evaluation is heavily weighted toward Amharic due to better dataset availability for that language compared to Tigrinya or Ge’ez.
  • Frequency vs. Linguistics: The reduction rules are learned statistically from corpus frequency. While they often align with actual morphemes, they aren't strictly linguistic. Incorporating a curated affix inventory could refine this.
  • Generative Models: This study focused on encoder-only models (like BERT). Future work is needed to see how these reversible schemes affect the performance of generative (decoder) models.
  • Broader Abugidas: The FIDEL approach could potentially be adapted for other abugida scripts, such as those used in South Asia (e.g., Devanagari).

For more details, you can find the full paper: "FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages" (arXiv:2611.20216v1).

Summary updated

Abstract

Subword tokenizers assume that meaningful linguistic units can be recovered by splitting text at learned boundaries, an assumption better suited to concatenative morphology than to Semitic root-and-pattern systems. The challenge is acute for the Ethiopic languages, whose abugida script encodes consonant-vowel units as single characters and hides consonantal structure. We introduce FIDEL (FIdäl DEcomposition and Linearization), a family of reversible abugida-aware pre-tokenization schemes that expose consonant and vowel-order structure, extend SPLINTER reductions from abjad scripts to the Ethiopic abugida, and evaluate them on three Ethiopic languages under controlled conditions, holding corpus, architecture, budget and protocol fixed so that tokenization is the only variable. We report intrinsic tokenizer statistics for BPE and UnigramLM across vocabulary sizes alongside downstream NLP tasks, including transfer scenarios. We find that FIDEL reduction improves news classification consistently and named-entity recognition at larger capacity, while sentiment analysis performance declines under the more aggressive schemes, and extractive question answering performance is preserved.
BibTeX
@misc{reda2026fidel,
  author = {Gebreslassie Teklu Reda and Yuval Pinter},
  title = {{FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages}},
  year = {2026},
  eprint = {2611.20216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20216},
}
Figure