TL;DR
This paper introduces the Jordan Comprehensive Contemporary Arabic Corpus (JCCA), a 100-million-word resource designed to be a balanced, representative, and annotated corpus of modern Arabic. It aims to fill a critical gap in available Arabic language resources, particularly for contemporary spoken and written varieties, by providing a comprehensive, open-source dataset comparable to established corpora like the British National Corpus.
What Problem Does This Paper Address?
Creating comprehensive and representative language corpora is crucial for advancing natural language processing (NLP) research. While numerous Arabic corpora exist, many suffer from limitations: they are often heavily skewed towards specific genres (like news), focus on older forms of Arabic (Classical Arabic), or lack detailed annotations. Furthermore, freely available, morphologically and syntactically annotated corpora for contemporary Arabic, especially those encompassing spoken dialects, are rare. This scarcity hinders research into modern Arabic linguistics, dialectology, and the development of robust Arabic NLP tools. The JCCA project directly addresses this by aiming to build a large-scale, balanced, and richly annotated corpus that reflects the diversity of contemporary Arabic usage across various regions and registers.
Methods
The JCCA was constructed following the model of the British National Corpus (BNC) to ensure balance and comprehensiveness. The corpus comprises 87% written texts and 13% transcribed spoken language, drawn from a wide range of genres and domains. A key methodological innovation lies in the annotation process. The authors adapted and extended the DIWAN tool, a dialectal word annotation tool, by developing a new tag-set based on traditional Arabic grammar. This enhanced tag-set includes detailed morphological and syntactic information, as well as the identification of word roots and morphological patterns for nouns and verbs. Sentiment analysis was also incorporated. A one-million-word "gold standard" snapshot of the corpus was manually annotated and verified by 25 students and supervised by linguists to ensure high quality. For web-sourced texts, a "scrambling" technique was employed to circumvent copyright issues while preserving the data's utility.
Datasets
The primary dataset is the Jordan Comprehensive Contemporary Arabic Corpus (JCCA) itself, a 100-million-word collection. This corpus is designed to be a benchmark for contemporary Arabic. A one-million-word subset of the JCCA serves as a "gold standard" for manual annotation and verification. The paper also references numerous other Arabic corpora (e.g., BNC, OSAC, KACST Arabic Corpus, ArabiCorpus, KSUCCA, arTenTen, HAC, Quranic Arabic Corpus, Curras Palestinian Arabic corpus) as context for the existing landscape of Arabic language resources.
Key Insights and Ideas
- The need for dialectal representation: The paper strongly emphasizes the importance of including spoken dialects in a comprehensive contemporary Arabic corpus, arguing that they are essential for accurately reflecting current language use.
- Leveraging traditional grammar for annotation: The authors demonstrate the value of grounding modern NLP annotation schemes in established linguistic traditions (traditional Arabic grammar) to create more linguistically informed and accurate tags.
- DIWAN as a flexible annotation platform: The adaptation and extension of the DIWAN tool highlight its potential as a robust platform for detailed morphological and syntactic annotation of Arabic.
- "Scrambling" for copyright compliance: The innovative use of text "scrambling" to address copyright concerns for web-harvested data is a practical solution for corpus construction.
- Sentiment as a valuable annotation layer: The inclusion of sentiment annotation adds another dimension to the corpus, enabling research into the emotional tone of contemporary Arabic.
Key Results
The paper reports on the successful construction and annotation of the 100-million-word JCCA. The corpus is characterized by its balanced representation of written (87%) and spoken (13%) language, diverse genres, and geographical regions. A one-million-word manually annotated subset serves as a high-quality benchmark. The annotation process, enhanced by the new tag-set and DIWAN tool, provides detailed morphological, syntactic, and sentiment information.
Limitations and Future Work
The authors acknowledge that achieving perfect balance and representativeness is an ideal, and their approach involves approximation and scalability. Future work includes making the JCCA a "monitor corpus" by adding new texts annually to track language change, which will make it more amenable to lexicography. The paper also implicitly suggests that further research could explore the application of the JCCA for tasks like genre identification, text classification, and text data mining.
Summary updated