A Linguistic Method into Stemming of Arabic for Data Compression

Hussein Soori, Jan Platos, Václav SnáselDATESO 2013

TL;DR

This paper introduces a novel linguistic approach to stemming Arabic text, aiming to improve data compression by reducing word redundancy. By leveraging an online morphological parser and a set of carefully crafted rules, the authors demonstrate a promising method for accurately identifying and reducing Arabic words to their root forms, paving the way for more efficient text processing.

What Problem Does This Paper Address?

The Arabic language, being the sixth most used language globally and possessing a complex morphological structure, presents significant challenges for natural language processing (NLP) tasks like data compression and information retrieval. Unlike many other languages, Arabic has an agglutinative nature, where letters link together, and its script is written right-to-left. Furthermore, the presence of short vowels and diacritics, which are often omitted in everyday text, can drastically alter a word's meaning and part of speech. Existing stemming methods often struggle with these complexities, leading to inaccurate reductions and a loss of information. This paper aims to address this gap by developing a more linguistically informed stemming method that accounts for Arabic's unique characteristics.

Methods

The core of this paper's methodology lies in a two-pronged approach:

  1. Linguistic Parsing: The authors utilize the Stanford Online Parser, a powerful tool capable of parsing Arabic text and identifying 27 different Parts of Speech (POS). This parser is crucial for understanding the grammatical role of words, which is essential for accurate stemming.
  2. Rule-Based Stemming: Based on the POS tags provided by the parser, the paper defines a set of specific stemming rules for three main POS categories: nouns, verbs, and adjectives. These rules are designed to systematically remove prefixes, suffixes, and other morphological elements to arrive at a root or stem. The rules are implemented in a step-by-step manner for each POS and sub-POS, ensuring a structured and comprehensive reduction process. For instance, rules for nouns address determiners, singular and plural common nouns, and proper nouns. Verb rules differentiate between perfect, passive, and imperfect tenses. Adjective rules focus on removing determiners.

Datasets

The paper mentions using "news articles, from the BBC Arabic and Al Jazeera Arabic news portals" for their experiments. These articles were parsed using the Stanford Online Parser. The specific size or composition of these datasets is not detailed, but they serve as the basis for evaluating the proposed stemming rules.

Key Insights and Ideas

  • The Power of POS Tagging: The paper strongly emphasizes that accurate stemming in Arabic is heavily reliant on correctly identifying the Part of Speech of a word. Without this, rules designed to remove specific affixes can incorrectly alter words that share similar letter patterns but have different grammatical functions (e.g., removing a prefix 'ي' from verbs but also from proper nouns like 'Yemen').
  • Handling Ambiguity: The authors acknowledge that Arabic's rich morphology can lead to ambiguity. Their approach of using a parser to disambiguate POS is a key strategy to mitigate this.
  • Step-by-Step Rule Application: The structured, step-by-step application of stemming rules for each POS is a methodical way to handle the layered complexity of Arabic morphology. This systematic approach helps ensure that multiple morphological features are addressed sequentially.
  • The Challenge of Proper Nouns: The paper highlights a specific challenge where rules intended for common nouns (like removing the definite article 'ال') can incorrectly affect proper nouns where 'ال' is an integral part of the name (e.g., 'Germany'). This underscores the need for fine-grained rules.

Key Results

The paper reports generally positive results for their stemming rules:

  • Nouns: The rules for DTNNP (determiner + singular proper noun) and DTNN (determiner + singular common noun) produced "very good results." For DTNNS (determiner + plural common noun), more rules were needed to handle plural and dual suffixes, but NNPS (common noun, plural or dual) also yielded "very good results."
  • Verbs: The rules for VBD (perfect verb) and VBN (passive verb) produced "good results." However, VBP (imperfect verb) showed "a few bad results," indicating a need for enhancement.
  • Adjectives: The rules for DTJJ (determiner + adjective) were highly successful, with "almost all rules made for adjectives produced successful results."
  • Overall: The authors conclude that their rules produced "very good results" overall, though they acknowledge that most errors occurred in the VBP category.

Limitations and Future Work

The authors explicitly acknowledge several limitations and suggest future directions:

  • Limited POS Coverage: The current rules are set for only three main POS categories (nouns, verbs, and adjectives), out of the 27 identified by the Stanford Online Parser.
  • Errors in VBP: The imperfect verb (VBP) category showed a notable number of errors, suggesting that the rules for this category need further improvement and enhancement.
  • Parser Errors: The paper notes that some errors occurred due to the Stanford parser itself wrongly parsing certain words (e.g., "بالكلاب").
  • Vocabulary and Corpus Size: The authors suggest that future work should include testing the rules against a wider variety of vocabulary and larger corpora to ensure robustness and generalizability.
  • Inclusion of More POSs: Expanding the rule set to cover more POS categories is a clear next step.
  • Enhanced Rules: The overall evaluation indicates that the rules need to be improved and enhanced, particularly for categories that showed less successful results.

Summary updated

BibTeX
@inproceedings{soori2013linguistic,
  author = {Hussein Soori and Jan Platos and Václav Snásel},
  title = {{A Linguistic Method into Stemming of Arabic for Data Compression}},
  booktitle = {DATESO},
  year = {2013},
}
Figure