Abstract
Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that face word segmentation challenge. However, unlike other Asian languages, word segmentation in Urdu not only has space omission errors but also space insertion errors. This paper discusses how orthographic and linguistic features in Urdu trigger these two problems. It also discusses the work that has been done to tokenize input text. We employ a hybrid solution that performs an n-gram ranking on top of rule based maximum matching heuristic. Our best technique gives an error detection of 85.8% and overall accuracy of 95.8%. Further issues and possible future directions are also discussed.
Introduction
In most Western languages, a space is a reliable delimiter for words. In many Asian languages, such as Thai, spaces are not used at all. Urdu presents a unique challenge, sitting in a middle ground where spaces are used inconsistently as word boundaries.
Urdu utilizes the Nastalique script, characterized by context-sensitive character shaping. Characters are classified into two groups: joiners, which connect to subsequent characters, and non-joiners, which only connect to the character preceding them. This orthographic property leads to two primary issues for Natural Language Processing (NLP):
- Space Omission: When a word ends in a non-joiner, the subsequent word appears visually separate even without an intervening space. Consequently, writers frequently omit the space, leading computational systems to treat two distinct words as a single token.
- Space Insertion: Conversely, writers often insert "extra" spaces to force a character into its correct visual form (e.g., to prevent a prefix from joining a stem), causing a single semantic word to be split into multiple tokens.
Standard tokenization methods fail because the space in Urdu often serves as a formatting tool rather than a semantic delimiter. In fact, research suggests that up to 70% of Urdu spelling errors are actually space-related.
Methodology
The proposed system employs a multi-stage hybrid pipeline designed to address both space omission and insertion errors.
Pre-processing and Tokenization
The process begins with normalization and the removal of diacritics. The system then performs initial tokenization based on existing spaces and punctuation to create "Orthographic Words" (OWs). This step manages computational complexity by limiting the scope of subsequent analysis.
Space Omission Module
This module uses a rule-based "maximum matching" heuristic to generate potential segmentations for each OW. These candidates are then ranked using unigram and bigram probabilities derived from a training corpus.
Space Insertion Module
To handle space insertion, the system addresses specific linguistic phenomena:
- Reduplication: An edit distance algorithm identifies repeated or slightly varied word pairs (e.g., "slowly slowly").
- Abbreviations: A finite automaton recognizes transliterated English abbreviations, such as "PhD".
- Affixation & Compounding: The system performs lexical look-ups against a specialized dictionary of 1,850 compound words and a comprehensive list of common Urdu affixes.
Data and Evaluation
The n-gram models were trained on a manually segmented corpus of 70,000 words. Evaluation was conducted on a test corpus of 2,367 words, which included 404 specific segmentation errors (221 omissions and 183 insertions) sourced from news sites like BBC Urdu and Jang.
Results and Discussion
The hybrid system achieved a peak overall accuracy of 95.8% using unigram ranking. The unigram model was exceptionally effective at resolving space omissions, with an error detection rate of 96.8%.
Space insertion proved more challenging. While the system effectively identified affixes (89.4% accuracy), it struggled with compound words, achieving only a 44.4% detection rate. This difficulty stems from the often arbitrary nature of compounding in Urdu and the lack of consensus among native speakers regarding word boundaries. A survey of 30 native speakers confirmed that human disagreement is common, particularly regarding compound words and verb auxiliaries, suggesting that "perfect" segmentation may be subjective.
Conclusion
While the current system shows strong performance, several areas for improvement remain. The model currently struggles with Out-of-Vocabulary (OOV) words, such as foreign names, which it may split into nonsensical morphemes. Future work will involve testing on larger, more diverse datasets and incorporating Part-of-Speech (POS) tagging to better disambiguate standalone words from those that are part of compounds or affixes.
Summary updated