LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual

Mona T. Diab, Nizar Habash, Owen Rambow, Ryan RotharXiv 2013 · arXiv

Introduction

In the field of Natural Language Processing (NLP), the ability to compare results across different studies is fundamental to scientific progress. For researchers working with Arabic, the Linguistic Data Consortium (LDC) provides high-quality annotated corpora, such as the Arabic Treebanks (ATB). However, because these corpora are typically released as bulk collections, research groups have historically defined their own training, development, and test splits. This lack of standardization created a "Wild West" scenario where performance metrics like F1-scores were not directly comparable because models were evaluated on different subsets of data. This technical report establishes a standardized, deterministic protocol for splitting ATB corpora to ensure reproducibility and "apples-to-apples" comparisons.

The 10-80-10 Protocol

The authors propose a volume-based procedure to create consistent divisions across all current and future Arabic treebanks. The protocol relies on a simple "10-80-10" rule designed to be deterministic and document-aware:

  1. Alphabetical Sorting: All documents within a treebank are sorted by filename. This provides a fixed, reproducible order without requiring complex metadata.
  2. Development Set (DEV): Documents are selected from the beginning of the sorted list until the total word count exceeds 10% of the corpus.
  3. Testing Set (TEST): Documents are selected from the end of the sorted list until the total word count exceeds 10% of the corpus.
  4. Training Set (TRAIN): The remaining ~80% of the documents in the middle form the training set.

By taking the DEV and TEST sets from opposite ends of the sorted list, the authors maximize the "topical distance" between the sets. This ensures that high performance is a result of model robustness rather than the model simply encountering very similar news stories in both training and testing.

Comprehensive Coverage

The manual provides specific document ranges for a wide array of LDC corpora, spanning Modern Standard Arabic (MSA), Egyptian Arabic (from the BOLT project), and Levantine Arabic.

Table 1: Overview of MSA, Egyptian, and Levantine Treebanks covered in the manual.

Beyond simple list-making, the report addresses complex data scenarios. For instance, ATB6 is identified as a special case because it contains a mix of genres (newswire, newsgroups, and weblogs). The authors argue that these genres must be treated as independent sub-treebanks with their own 10-80-10 splits to prevent cross-genre contamination. Additionally, the new TEST sets were intentionally designed as supersets of common legacy splits (such as those by Zitouni or MADA), allowing researchers to maintain backward compatibility with older literature.

Standardized Results

The primary contribution of this work is the definitive set of tables listing exact document ranges, document counts, and word counts for every major Arabic treebank. These tables serve as the "ground truth" for Arabic NLP evaluation.

Table 2: Standardized data divisions for ATB1, ATB2, and ATB3.

As shown in the data for ATB1, the standard now dictates a DEV set of 71 documents (14,635 words) and a TEST set of 95 documents (14,553 words). By basing splits on word volume rather than file count, the authors account for the significant variance in document lengths, ensuring that test sets are neither too small nor unexpectedly large.

Conclusion

By moving away from ad-hoc or random splits, this manual provides a unified framework for the Arabic NLP community. While alphabetical sorting is a heuristic, its reproducibility and the separation of DEV and TEST sets provide a rigorous foundation for future research. As the LDC continues to release new data, this protocol offers a clear path for extending these standards to maintain a consistent evaluation environment.

Summary updated

Abstract

The Linguistic Data Consortium (LDC) has developed hundreds of data corpora for natural language processing (NLP) research. Among these are a number of annotated treebank corpora for Arabic. Typically, these corpora consist of a single collection of annotated documents. NLP research, however, usually requires multiple data sets for the purposes of training models, developing techniques, and final evaluation. Therefore it becomes necessary to divide the corpora used into the required data sets (divisions). This document details a set of rules that have been defined to enable consistent divisions for old and new Arabic treebanks (ATB) and related corpora.
BibTeX
@misc{diab2013ldc,
  author = {Mona T. Diab and Nizar Habash and Owen Rambow and Ryan Roth},
  title = {{LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual}},
  year = {2013},
  eprint = {1309.5652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1309.5652},
}
Figure