Annotating Targets of Opinions in Arabic using Crowdsourcing

Noura Farra, Kathy McKeown, Nizar Habash

Abstract

Researchers from Columbia University and NYU Abu Dhabi have developed a robust, two-stage crowdsourcing pipeline to identify "targets of interest" and their associated sentiment in Arabic text. By decomposing a complex linguistic task into simpler sub-steps on Amazon Mechanical Turk, the team created a high-quality dataset of over 4,300 opinion targets with a remarkable 91.8% annotator agreement. This work addresses the scarcity of fine-grained sentiment resources for Arabic and demonstrates the efficacy of crowdsourcing for complex linguistic annotation.

Introduction

Most sentiment analysis tools identify whether a sentence is generally positive or negative, but they often fail to pinpoint the specific entity toward which the sentiment is directed—a task known as fine-grained subjectivity analysis. For instance, in the sentence "The people will prevail over the regime’s brutality," a system must recognize positive sentiment toward "the people" and negative sentiment toward "the regime."

This task is particularly challenging in Arabic due to several factors:

  1. Boundary Ambiguity: Annotators often disagree on where a target span starts and ends (e.g., "the regime" vs. "the regime's brutality").
  2. Linguistic Complexity: Arabic's rich morphology, where prepositions and conjunctions (clitics) are attached directly to words, complicates span identification.
  3. Resource Scarcity: While English has numerous gold-standard datasets for fine-grained sentiment, Arabic has historically lacked publicly available resources for opinion target annotation.

Methodology

The researchers designed a two-stage workflow using Amazon Mechanical Turk (AMT) to make the task manageable for non-expert workers:

  • Stage 1: Entity Identification: Three annotators identified main nouns and noun phrases. To resolve boundary disagreements, the researchers used an "intersection" method, retaining only entities mentioned by all three workers and selecting the "maximal phrase" (the longest version of the agreed-upon span).
  • Stage 2: Sentiment Assignment: For each identified entity, five new annotators assigned a polarity: positive, negative, or neutral. An entity was confirmed as a "target" if at least three out of five workers agreed on a positive or negative polarity.

To ensure quality, the study employed qualification tasks to filter for native Arabic speakers and provided instructions in Modern Standard Arabic (MSA).

Examples of annotated targets and their polarities across different domains.

Dataset Characteristics

The study utilized 1,177 comments from the Qatar Arabic Language Bank (QALB), which contains online commentaries on Aljazeera news articles. To ensure a balanced sentiment profile, the authors used Latent Dirichlet Allocation (LDA) topic modeling to select comments from three distinct domains:

  • Politics: 596 comments (predominantly negative).
  • Culture: 382 comments (mixed sentiment).
  • Sports: 199 comments (predominantly positive).

The resulting dataset contains 7,346 entities, 4,345 of which were confirmed as targets of polar opinions.

Key Results and Analysis

The crowdsourcing pipeline achieved high data quality, with a 91.8% majority agreement on entity-level sentiment. The final dataset revealed clear domain-specific sentiment trends, as shown in the table below.

Distribution of positive and negative sentiment targets by domain.

Key findings include:

  • Sentiment Distribution: 70% of political targets were negative, whereas 79% of sports targets were positive, justifying the use of topic modeling to balance the dataset.
  • Target Complexity: While most targets (38%) were two words long, 10% were complex phrases of five or more words, highlighting the difficulty of span identification.
  • Linguistic Challenges: Arabic morphology presented unique issues; clitics (like "and" or "to") were often included in target spans, requiring careful tokenization in future modeling. Additionally, "verbless" nominal sentences occasionally led annotators to confuse complex noun phrases with full sentences.

Conclusion

The success of this project highlights that complex linguistic annotation can be effectively crowdsourced by reducing the cognitive load on workers—specifically by separating entity identification from sentiment judgment. The researchers have made the corpus publicly available, providing a valuable benchmark for the Arabic NLP community. Future work will focus on co-reference resolution (linking repeated mentions of the same target) and explicitly mapping targets to the specific words that express the associated sentiment.

Summary updated

BibTeX
@inproceedings{farra2015annotating,
  author = {Noura Farra and Kathy McKeown and Nizar Habash},
  title = {{Annotating Targets of Opinions in Arabic using Crowdsourcing}},
  booktitle = {ANLP @ ACL},
  year = {2015},
  doi = {10.18653/v1/w15-3210},
}
Figure