MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages

Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang NejdlEMNLP 2025

TL;DR

This paper introduces MoVoC, a novel approach to subword tokenization that explicitly incorporates morphological information for Ge'ez script languages. By creating morphology-aware subword vocabularies and a specialized tokenizer (MoVoC-Tok), the authors aim to improve linguistic fidelity and token efficiency in morphologically rich, low-resource languages, demonstrating improved intrinsic metrics and promising downstream translation performance.

What Problem Does This Paper Address?

Modern NLP models heavily rely on subword tokenization, like Byte Pair Encoding (BPE), to handle large vocabularies and out-of-vocabulary words. However, standard subword methods often fail to respect morphological boundaries, especially in morphologically complex languages. This is particularly problematic for low-resource languages, where such languages often receive fewer subwords, leading to inefficient representations and potential loss of linguistic meaning. Ge'ez script languages, with their rich inflectional morphology and limited annotated data, exemplify this challenge, making them difficult to process effectively with existing tokenization techniques.

Methods

The authors propose MoVoC (Morpheme-Aware Vocabulary Construction), a hybrid approach that combines linguistically informed morpheme segmentation with standard BPE. The process involves:

  1. Pre-tokenization and Supervised Morphological Analysis: Raw text is cleaned, and then a supervised morphological analysis is performed. For Amharic and Tigrinya, they leverage the HornMorpho tool. For Ge'ez and Tigre, due to a lack of existing analyzers, they manually annotate morphemes under linguistic supervision.
  2. Vocabulary Construction (MoVoC): A hybrid vocabulary is created by allocating a predefined portion to morpheme units and the rest to BPE tokens. This balances linguistic structure with statistical efficiency, aiming to reduce rare tokens and preserve semantic granularity.
  3. Morpheme-aware Subword Segmentation (MoVoC-Tok): A BPE tokenizer is trained using the MoVoC vocabulary. Crucially, morphological constraints are integrated into the BPE training process, limiting merge operations to those that do not cross morpheme boundaries defined by MoVoC. This ensures that the resulting tokens adhere to true morphological segmentation.

Datasets

  • Training Data:
    • HornMT3 corpus (for Amharic and Tigrinya)
    • NLLB project parallel corpora (English-Tigrinya, English-Amharic, and English-Tigre for training BPE vocabularies)
  • Evaluation Data:
    • Flores-200 development and test sets (for Amharic and Tigrinya)
    • OPUS parallel corpus (for Amharic, Tigrinya, Tigre, and Ge'ez, including newly created and validated pairs for Ge'ez)
    • Manually annotated morpheme test sets for intrinsic evaluation.
  • Newly Introduced Benchmarks: The authors created and released manually annotated morpheme datasets for four Ge'ez script languages (Amharic, Tigrinya, Ge'ez, and Tigre) and morpheme-aware vocabularies for two of them. These serve as a benchmark for morphological segmentation evaluation.

Key Insights and Ideas

  • Morphological Awareness is Crucial for Low-Resource Languages: Standard subword tokenizers, while effective in high-resource settings, fail to capture the nuances of morphologically rich languages, especially when data is scarce. MoVoC demonstrates that explicitly incorporating morphology can bridge this gap.
  • Hybrid Approach Outperforms Pure BPE: Merging morpheme-based tokens with BPE tokens, and constraining BPE merges, leads to more linguistically meaningful units than BPE alone.
  • Intrinsic Metrics Reveal Deeper Improvements: While automatic translation metrics (like BLEU) show modest gains, intrinsic metrics like MorphScore and Boundary Precision significantly improve, highlighting the success of MoVoC in preserving morphological integrity.
  • Ge'ez Script Languages as a Challenging Testbed: The inherent complexity of Ge'ez script languages, with their fusional morphology and agglutination, makes them ideal for testing and demonstrating the effectiveness of morphology-aware tokenization.

Key Results

  • Intrinsic Evaluation: MoVoC-Tok achieved substantially higher MorphScores and improved Morpheme Boundary Precision and Rényi Entropy across all four target languages compared to standard BPE and WordPiece. For example, Tigre saw a significant improvement in MorphScore from 74.6% (BPE) to 83.9% (MoVoC-Tok).
  • Downstream Task Performance (Machine Translation): MoVoC-Tok consistently outperformed BPE and WordPiece in BLEU and chrF++ scores for English-to-Amharic, English-Tigrinya, and English-Tigre translation tasks. For instance, English-to-Tigre translation saw an increase in BLEU from 0.1025 (WordPiece) to 0.1175 (MoVoC-Tok).
  • Qualitative Analysis: Examples showed MoVoC preserving morphological integrity (e.g., correctly segmenting negation prefixes) and leading to more accurate translations by providing linguistically meaningful units.

Limitations and Future Work

  • Limited Gains in Automatic Translation: The authors acknowledge that MoVoC does not yield significant gains in automatic translation quality, suggesting that while tokenization is improved, other factors might limit downstream performance.
  • Dataset Size and Generalizability: The curated datasets and vocabularies are limited to a small set of Ge'ez script languages, which may affect the generalizability of the method to other languages or broader NLP tasks.
  • Complexity of Hybrid Approach: The increased complexity of the hybrid tokenization approach might not always translate to proportional performance improvements in all downstream NLP tasks.
  • Future Work: The authors plan to release their annotated datasets and tokenizer to support further research in low-resource, morphologically rich languages. They also suggest that future work could explore more sophisticated methods for integrating morphological information into tokenization.

Summary updated

BibTeX
@inproceedings{teklehaymanot2025movoc,
  author = {Hailay Kidu Teklehaymanot and Dren Fazlija and Wolfgang Nejdl},
  title = {{MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure