Proposed Framework for the Evaluation of Standalone Corpora Processing Systems: An Application to Arabic Corpora

Abdulmohsen Al-Thubaity, Hend S. Al‐Khalifa, Reem Alqifari, Manal AlmazruaThe Scientific World JOURNAL 2014 · DOI

TL;DR

This paper introduces a novel, comprehensive framework for evaluating standalone corpus processing systems, focusing on usability, functionality, and performance, with a specific emphasis on Arabic language support. By applying this framework to seven freely available systems, the authors reveal significant disparities in their capabilities, particularly concerning Arabic language processing and N-gram analysis, offering valuable guidance for users and developers alike.

What Problem Does This Paper Address?

In the fields of Natural Language Processing (NLP), corpus linguistics, and language learning, researchers and students often need to process and analyze their own custom corpora. While numerous online corpora and processing tools exist, they may not always cater to specific needs, especially for less common languages or specialized linguistic phenomena. Existing evaluations of corpus processing systems have often been fragmented, focusing on limited aspects or lacking a holistic approach. This paper addresses the need for a standardized, multi-dimensional framework to systematically evaluate these systems, considering not just their features but also how easy they are to use and how well they perform, with a crucial focus on their suitability for processing Arabic text.

Methods

The authors propose a three-pronged evaluation framework:

  • Usability: Assessed using a modified expert review checklist based on 161 heuristics, covering nine dimensions like "Main screen," "Task orientation," and "Help, feedback, and error tolerance." Each criterion was rated on a scale of -1 (does not comply), +1 (complies), or 0 (partially complies).
  • Functionality: Evaluated using 54 criteria across 10 dimensions, specifically designed to assess features relevant to corpus processing, including "User interface languages," "Input text format," "N-gram frequency lists," and "Concordance." A key aspect was evaluating the systems' ability to handle Arabic language specifics, such as writing direction and character normalization.
  • Performance: Measured by timing how long each system took to generate results for three core functionalities (single word frequency, 2-gram frequency, and concordance) using three distinct Arabic corpora of varying sizes.

Datasets

The performance evaluation utilized three Arabic corpora:

  • 2012 Corpus of Arabic Newspapers (2012 CAN): Approximately 2.2 million words.
  • KACST Arabic Text Classification Corpus (KACST ATCC): Split into two parts (4.3 million and 7.2 million words) for a total of 11.5 million words.
  • King Saud University Corpus of Classical Arabic (KSUCCA): Approximately 50.6 million words.

These datasets were chosen to represent different sizes and structures of Arabic text, allowing for a robust performance assessment.

Key Insights and Ideas

  • Usability is relatively uniform, but critical gaps remain: While most systems offered comparable usability scores (averaging 77%), a significant number were deficient in "help, feedback, and error tolerance" and "search" functionalities, hindering ease of use.
  • Functionality for Arabic is a major bottleneck: Many systems struggled with Arabic-specific processing, particularly in "preprocessing" (e.g., handling diacritics) and "collocation" extraction. Only two systems (aConCorde and Khawas) adequately considered Arabic language characteristics.
  • N-gram analysis is a weak point across the board: The generation of N-gram frequency profiles proved to be a significant challenge for most systems, indicating a general lack of robust support for this crucial NLP task.
  • Performance scales poorly with corpus size: As corpus size increased, system performance degraded significantly, with some systems even failing to process larger datasets. This highlights the need for efficient algorithms and robust memory management.
  • Khawas V3.0 emerges as a strong contender for Arabic: This system demonstrated superior performance and functionality, particularly in its handling of Arabic language specifics, including writing direction and normalization, and its ability to process larger corpora compared to many others.

Key Results

  • Usability: Scores ranged from 69% to 80%, with an average of 77%. AntConc V3.2.4 showed the highest usability, while TextStat V3.0 had the lowest.
  • Functionality: Scores varied widely, from 17% to 74%, with an average of 39%. Khawas V3.0 achieved the highest score (74%), while TextStat V3.0 scored the lowest (17%). Concordance and user interface languages were the strongest functionality criteria, while N-gram frequency generation was the weakest.
  • Performance: TextSTAT V2.9 generally outperformed other systems in speed, while KWIC V5.0 was the slowest. However, performance was highly dependent on corpus size, with many systems stopping or becoming unavailable for the largest corpus (KSUCCA). Khawas V3.0 was the only system capable of generating 2-gram frequency profiles for corpora up to 7 million words.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited scope of systems: Only seven systems were evaluated, leaving many other available tools unexamined.
  • Difficulty in direct statistical comparison: The diverse characteristics of the systems made in-depth statistical comparisons challenging.
  • Language-specific focus: The framework was applied only to Arabic, limiting generalizability without further testing.

Future work could involve:

  • Applying the framework to other languages, particularly well-studied ones like English, to compare results and identify broader trends.
  • Expanding the evaluation to include a wider range of commercial and free corpus processing systems.
  • Further refining the statistical analysis to allow for more direct comparisons between systems.
  • Investigating the impact of different hardware configurations on system performance.

Summary updated

Abstract

Despite the accessibility of numerous online corpora, students and researchers engaged in the fields of Natural Language Processing (NLP), corpus linguistics, and language learning and teaching may encounter situations in which they need to develop their own corpora. Several commercial and free standalone corpora processing systems are available to process such corpora. In this study, we first propose a framework for the evaluation of standalone corpora processing systems and then use it to evaluate seven freely available systems. The proposed framework considers the usability, functionality, and performance of the evaluated systems while taking into consideration their suitability for Arabic corpora. While the results show that most of the evaluated systems exhibited comparable usability scores, the scores for functionality and performance were substantially different with respect to support for the Arabic language and N-grams profile generation. The results of our evaluation will help potential users of the evaluated systems to choose the system that best meets their needs. More importantly, the results will help the developers of the evaluated systems to enhance their systems and developers of new corpora processing systems by providing them with a reference framework.
BibTeX
@article{althubaity2014proposed,
  author = {Abdulmohsen Al-Thubaity and Hend S. Al‐Khalifa and Reem Alqifari and Manal Almazrua},
  title = {{Proposed Framework for the Evaluation of Standalone Corpora Processing Systems: An Application to Arabic Corpora}},
  journal = {The Scientific World JOURNAL},
  year = {2014},
  doi = {10.1155/2014/602745},
}
Figure