Towards Responsible Natural Language Annotation for the Varieties of Arabic

A. Stevie Bergman, Mona T. Diab

Abstract

When building NLP models, there is a tendency to aim for broader coverage, often overlooking cultural and (socio)linguistic nuance. In this position paper, we make the case for care and attention to such nuances, particularly in dataset annotation, as well as the inclusion of cultural and linguistic expertise in the process. We present a playbook for responsible dataset creation for polyglossic, multidialectal languages. This work is informed by a study on Arabic annotation of social media content.

Introduction

Arabic is not a monolith; it is a family of diverse and often mutually unintelligible varieties. Most Natural Language Processing (NLP) systems are designed with a "one-size-fits-all" approach, typically focusing on Modern Standard Arabic (MSA). However, on social media, regional dialects differ as significantly as Spanish does from Romanian. Ignoring these nuances leads to systemic biases, where benign content is mislabeled as harmful due to a lack of local expertise, creating an "access disparity" for speakers of less-represented varieties.

Methodology

The study utilized a dataset of approximately 3,000 Facebook posts per country, covering 10 nations across the Maghrebi, Levantine, Iraqi, Egyptian, Peninsular/Gulf, and Yemeni regions. To quantify the impact of linguistic distance, the researchers employed Signal Detection Theory (SDT) to compare two groups of annotators:

  1. AiM (Annotators in Morocco): A cohort based in one specific region.
  2. CE (Country Experts): Native speakers with local knowledge specific to the content's origin.

Annotators labeled content as "delete" (for hate speech) or "ignore" (for benign content), allowing researchers to measure accuracy and "strictness" across dialectal boundaries.

Key Results and Insights

The Leniency of Understanding

A primary finding of the study is that linguistic competence directly correlates with annotation leniency. When annotators understand nuances like sarcasm or idiomatic speech, they are less likely to incorrectly flag benign content as hateful. Conversely, a lack of context leads to "trigger-happy" moderation.

Figure 1: Comparison of annotation behavior across different Arabic-speaking regions. The 'MA' (Morocco) group shows the highest alignment/leniency when annotating their own dialect, while accuracy and leniency fluctuate significantly for other regions like YE (Yemen).

The High Stakes of "Faux Amis"

False cognates, or faux amis, pose a significant risk in cross-dialectal moderation. For example, the word zamel is a slur in Moroccan Arabic but refers to a specific type of singing in Yemeni. Without local experts, a Moroccan-centric model would unfairly censor Yemeni cultural expression.

Dialectal Dominance

The study confirmed that local dialects, rather than MSA, dominate social media communication. Furthermore, geographic location (IP address) is an imperfect proxy for dialect; in the Saudi Arabian dataset, 74% of samples were identified as "non-Saudi" content, likely reflecting the country's high population of migrant workers.

The Playbook for Responsible Annotation

To address these disparities, the paper proposes an 8-point playbook for building culturally competent datasets:

  • Sample Routing: Direct content to experts of that specific variety.
  • Proficiency Assessments: Test annotators on regional dialects, not just formal MSA.
  • Nuanced Guidelines: Develop labeling criteria that account for regional slang and cultural context.
  • Reviewer Well-being: Ensure annotators are supported when dealing with sensitive content.

Conclusion

Models trained exclusively on high-resource varieties like MSA are fundamentally ill-equipped for the linguistic reality of the modern web. This work advocates for a shift toward "responsible annotation" that prioritizes local expertise. Future research will explore applying this playbook to other multidialectal language families, such as Indo/Malay or varieties of German, to ensure global NLP systems are equitable and culturally aware.

Summary updated

Abstract

When building NLP models, there is a tendency to aim for broader coverage, often overlooking cultural and (socio)linguistic nuance. In this position paper, we make the case for care and attention to such nuances, particularly in dataset annotation, as well as the inclusion of cultural and linguistic expertise in the process. We present a playbook for responsible dataset creation for polyglossic, multidialectal languages. This work is informed by a study on Arabic annotation of social media content.
BibTeX
@inproceedings{bergman2022towards,
  author = {A. Stevie Bergman and Mona T. Diab},
  title = {{Towards Responsible Natural Language Annotation for the Varieties of Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=ILnwAT8Jke},
}
Figure