Shamela: A Large-Scale Historical Arabic Corpus

Yonatan Belinkov, Alexander Magidow, Maxim Romanov, Avi Shmidman, Moshe Koppel

Abstract

Arabic is a widely-spoken language with a rich and long history spanning more than fourteen centuries. Yet existing Arabic corpora largely focus on the modern period or lack sufficient diachronic information. We develop a large-scale, historical corpus of Arabic of about 1 billion words from diverse periods of time. We clean this corpus, process it with a morphological analyzer, and enhance it by detecting parallel passages and automatically dating undated texts. We demonstrate its utility with selected case-studies in which we show its application to the digital humanities.

Introduction

While Arabic is a major world language with a literary history stretching back to the 7th century, most digital resources focus heavily on Modern Standard Arabic (MSA). This lack of "diachronic" metadata creates a significant gap for linguists, Digital Humanities (DH) scholars, and NLP researchers who require specialized data to build tools for classical and medieval texts. To address this, we introduce Shamela, a massive 1-billion-word historical corpus spanning over 14 centuries.

Methodology

The corpus was constructed by scraping and cleaning data from Al-Maktaba Al-Shamela, a popular digital library. The processing pipeline included:

  • Metadata Wrangling: Reconciling inconsistent author names and titles to ensure accurate attribution.
  • Lemmatization: Using the MADAMIRA morphological analyzer to reduce 16.8 million surface words to ~95,000 lemmas, facilitating deep semantic analysis.
  • Text Reuse Detection: Adapting a hash-based skip-gram algorithm to identify over 5 million parallel passages, accounting for the common historical practice of quotation and paraphrase.
  • Automatic Dating: Developing a ranking algorithm based on 5-gram language models to predict the era of ~1,200 undated texts with high accuracy.

Corpus Overview

The Shamela Corpus contains over 6,100 texts. Approximately 800 million words are associated with specific dates based on the author's death, providing a comprehensive timeline of the language's evolution.

Word and text counts per 50-year period, illustrating the temporal distribution and scale of the Shamela corpus.

Key Insights and Results

Linguistic Stability

A striking finding is the extreme stability of the Arabic language. While English words evolve rapidly, the corpus reveals that 90% of Arabic lemmas have been in use for over 700 years, and 80% for over 1,100 years. This explains why modern speakers can access ancient texts with relative ease compared to other languages.

Distribution of the lifespan of words in use, showing that the vast majority of Arabic lemmas persist for many centuries.

Debunking Linguistic Myths

The corpus allows researchers to challenge claims of "inauthenticity" in modern usage. For example, the word ḥawalay (meaning "around/approximately") was found in numerical contexts as early as 1201 CE, proving it is not a modern corruption as some linguistic "memes" suggest.

Dating Accuracy

The automatic dating algorithm achieved 42.95% accuracy at its top-1 prediction and over 70% accuracy within its top-3 predictions. This suggests that despite the language's stability, there are enough subtle shifts in vocabulary and syntax over the centuries for machine learning models to accurately estimate a text's age.

Conclusion and Future Work

Shamela provides a massive jump in available historical data, moving from previous datasets of ~50 million words to a full 1 billion words. While currently skewed toward religious and formal texts, it provides an unprecedented foundation for historical NLP. Future work will focus on refining language periodization and mapping the networks of mutual influence between historical scholars through text reuse analysis.

Summary updated

Abstract

Arabic is a widely-spoken language with a rich and long history spanning more than fourteen centuries. Yet existing Arabic corpora largely focus on the modern period or lack sufficient diachronic information. We develop a large-scale, historical corpus of Arabic of about 1 billion words from diverse periods of time. We clean this corpus, process it with a morphological analyzer, and enhance it by detecting parallel passages and automatically dating undated texts. We demonstrate its utility with selected case-studies in which we show its application to the digital humanities.
BibTeX
@inproceedings{belinkov2016shamela,
  author = {Yonatan Belinkov and Alexander Magidow and Maxim Romanov and Avi Shmidman and Moshe Koppel},
  title = {{Shamela: A Large-Scale Historical Arabic Corpus}},
  booktitle = {LT4DH @ COLING},
  year = {2016},
  url = {https://openreview.net/forum?id=mIFRvkYsYw},
}
Figure