TL;DR
This paper introduces OSIAN, a large, open-source Arabic news corpus containing 3.5 million articles and 1 billion tokens, annotated with lemmas and part-of-speech tags. It addresses the scarcity of high-quality Arabic linguistic resources by leveraging web crawling and integrating into the CLARIN infrastructure for broader accessibility and research.
What Problem Does This Paper Address?
The Arabic language, despite being widely spoken and present online, has historically been an "under-resourced" language in terms of computational linguistics. This means there's a significant lack of freely available, annotated corpora that are essential for developing and evaluating Natural Language Processing (NLP) and Information Retrieval (IR) systems for Arabic. While some progress has been made, building large, annotated corpora is expensive and time-consuming. This paper tackles this gap by creating a substantial, openly accessible Arabic news corpus, making it easier for researchers to work on Arabic NLP tasks.
Methods
The authors employed a methodology inspired by the Leipzig Corpora Collection (LCC) for building their corpus. The process involved:
- Web Crawling: Using Heritrix, a web crawler from the Internet Archive, they downloaded approximately six million webpages from international Arabic news broadcasting platforms.
- Data Cleaning and Filtering: Raw HTML/XML files were processed to extract plain text, remove HTML tags, and separate sentences.
- Annotation: A language-independent tool, Treetagger, was adapted and retrained using the Frequency Dictionary of Arabic (Buckwalter and Parkinson, 2014) to automatically annotate each word with its lemma and part-of-speech (PoS) tag. This adaptation and retraining is a key methodological contribution.
- Integration: The processed corpus was then integrated into the CLARIN (Common Language Resources and Technology Infrastructure) framework, making it discoverable and accessible through various CLARIN services like OAI-PMH for metadata, direct download of plain text (under CC BY-NC 4.0), and integration with the WebLicht annotation platform and CLARIN's Federated Content Search (FCS).
Datasets
The primary dataset introduced in this paper is the OSIAN (Open Source International Arabic News) Corpus.
- Source: Collected from 31 different international Arabic news broadcasting platforms freely available on the Web.
- Size: Approximately 3.5 million articles, containing over 37 million sentences and roughly 1 billion tokens.
- Annotation: Each article is annotated with metadata, and each word is tagged with its lemma and part-of-speech.
- Relation to Existing Datasets: OSIAN builds upon the need for more comprehensive Arabic resources, addressing limitations of previous efforts by providing a larger, more consistently annotated, and openly accessible dataset. It is distinct from raw corpora like OSAC by offering linguistic annotations and from partially tagged corpora like arTenTen by providing full lemma and PoS tagging.
Key Insights and Ideas
- The Power of Web Crawling for Under-Resourced Languages: The paper demonstrates that systematic web crawling, combined with robust processing and annotation pipelines, can effectively create valuable linguistic resources for languages that have historically lacked them.
- Importance of Consistent Annotation: By adapting and retraining Treetagger, the authors ensured a consistent lemma and PoS annotation across a large corpus, which is crucial for reliable NLP research.
- CLARIN Integration as a Catalyst: Integrating OSIAN into the CLARIN infrastructure is a strategic move to ensure its long-term preservation, discoverability, and widespread use by the research community, bridging the gap between resource creation and utilization.
- Arabic Word Length Distribution: The analysis of word length distribution reveals that a significant percentage of words in Arabic news articles are longer than average, which has implications for stemming and lemmatization techniques.
Key Results
- The OSIAN corpus contains approximately 1 billion tokens derived from 3.5 million articles.
- The Treetagger, after adaptation and retraining, achieved an accuracy rate of 95.02% on a manually checked sample of 10,000 words.
- The corpus is encoded in XML, with each article containing metadata and lemmatized, PoS-tagged sentences.
- The data is made available under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
Limitations and Future Work
Limitations:
- The paper notes that while some URLs include topic information, many do not, leading to "unknown" topic labels for those articles.
- The number of extracted articles varies significantly across different websites, influenced by factors like crawling duration, website restrictions, and external links.
Future Work:
- Corpus Expansion: Extend OSIAN to cover more international Arabic news, incorporating diachronic (historical) and geographical perspectives to enable studies on language change and variation.
- Tool Improvement: Aim to improve the accuracy of the annotation tools and explore new, meaningful forms of annotation.
- Enhanced CLARIN Integration: Support FCS 2.0 and querying of annotation layers, and explore the use of the OSIAN corpus within corpus linguistics, ANLP, and AIR research areas.
- Advanced Querying: Make the corpus available via the KonText advanced corpus query interface for better compatibility with FCS-QL specifications.
Summary updated