TL;DR
This paper investigates how using multiple writing systems (Perso-Arabic and Latin scripts) and varying model capacity impacts subword tokenization for Soranî Kurdish, a morphologically rich but under-resourced language. They find that joint training across scripts offers benefits, particularly for the Unigram tokenizer, and that shorter subwords generally lead to better morphological alignment.
What Problem Does This Paper Address?
Tokenization, the process of breaking down text into meaningful units, is a fundamental challenge in Natural Language Processing (NLP), especially for morphologically complex languages. Soranî Kurdish, spoken in Iran and Iraq, presents a unique challenge because it's written in two distinct scripts: a modified Perso-Arabic script and a Latin-based script. These scripts have different structural properties, which can affect how language models learn and segment words. Existing research often focuses on single scripts or languages with more abundant resources. This paper aims to understand if combining data from both Soranî scripts and adjusting model capacity can lead to better subword tokenization, which is crucial for downstream NLP tasks like machine translation and text analysis, particularly for low-resource languages.
Methods
The researchers explored two primary subword tokenization algorithms: Byte Pair Encoding (BPE) and Unigram tokenization. BPE works by iteratively merging the most frequent character sequences, while Unigram tokenizes by optimizing the probability of the training data, allowing for more flexible splits. They investigated the impact of:
- Script Diversity: Training models on either the Latin script, the Arabic script, or a combined "joint" corpus of both.
- Model Capacity: Varying the vocabulary size of the subword models (e.g., 2514, 5000 subwords).
The core idea is to see if "data augmentation" through script diversity, combined with appropriate model capacity, can improve the models' ability to capture the morphological structure of Soranî Kurdish. They evaluated tokenization quality by measuring how well the generated subwords align with actual morpheme boundaries.
Datasets
The primary dataset used was the Asosoft corpus, a collection of 188 million word tokens and 4.66 million word types for Soranî Kurdish. This corpus was preprocessed to remove newline characters, repeated characters, and redundant whitespace. For evaluation, a manually tokenized test set of 1500 Soranî words was created, representing various linguistic phenomena and script-specific features like ezafe constructions and half spaces.
Key Insights and Ideas
- Joint Training Benefits: Combining data from both Latin and Arabic scripts for training generally improved tokenization quality, especially for the Unigram model. This suggests that leveraging script diversity can indeed enhance morphological segmentation.
- Unigram's Strength: Unigram tokenization consistently outperformed BPE across most measures of morphological structure, aligning with previous research that suggests its probabilistic approach is better suited for capturing nuanced linguistic patterns.
- Shorter Subwords, Better Morphemes: The study found that shorter subwords, often a result of BPE's merge-based procedure or smaller vocabulary sizes, tend to map more closely to actual morphemes. This indicates that finer-grained tokenization can be beneficial for morphological analysis.
- Vocabulary Size vs. Data Augmentation: For single-script models, increasing vocabulary size seemed to be a more effective strategy for improving performance than simply using more data from that single script.
Key Results
- Tokenization Consistency: Larger, independently trained BPE models achieved the highest token match rate (79.67%) between Latin and Arabic scripts, indicating good consistency in how they segment words across different writing systems.
- Morphological Accuracy: Unigram tokenization, particularly when trained jointly, showed the highest percentage of syllable-token correspondence. Table 2 shows that Unigram models generally achieved higher Morpheme Coverage and Segmentation Accuracy percentages compared to BPE models, especially for the joint training scenarios. For instance, joint Unigram training achieved 54% coverage and 43% accuracy on the Arabic script, outperforming BPE.
- Subword Length: BPE models, especially smaller ones, produced shorter subwords (average token length around 2.87-3.46) compared to Unigram models (average token length around 3.09-3.33 for joint, and up to 5.09 for single script).
Limitations and Future Work
The authors acknowledge that their study primarily focuses on unsupervised subword tokenizers. They suggest future work could involve comparing their findings with traditional unsupervised segmenters like Morfessor and DPSeg. Additionally, they propose exploring the effects of smaller training datasets with less frequency bias, building tokenization models based on vocabularies rather than entire corpora, and investigating greedy contextual decoding tokenizers. The paper also highlights the potential of leveraging script-specific orthographic cues, such as the Zero-width non-joiner (ZWNJ) in the Perso-Arabic script, to further improve tokenization models.
Summary updated