L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language

Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, Halima AlshabaniACL 2019

TL;DR

This paper introduces L-HSAB, the first publicly available dataset for detecting hate speech and abusive language specifically in the Levantine dialect of Arabic on Twitter. By carefully collecting and annotating tweets, the authors aim to provide a crucial benchmark for developing more accurate and nuanced detection systems for underrepresented Arabic dialects.

What Problem Does This Paper Address?

The proliferation of hate speech and abusive language on social media poses a significant challenge, with the potential to incite real-world harm. While research into detecting such content has grown, it has largely focused on Indo-European languages. Arabic, with its diverse dialects and informal nature, presents unique hurdles for automated detection. Existing Arabic datasets are often limited in scope, dialect coverage, or lack rigorous annotation evaluation. This paper specifically addresses the gap in resources for the Levantine dialect (Syrian, Lebanese, Palestinian, Jordanian), which, despite not being the most used on Twitter, is prevalent in politically charged discussions, making it a fertile ground for toxic content.

Methods

The authors developed the L-HSAB dataset through a multi-stage process. First, they collected tweets using keywords related to entities often targeted by hate speech (e.g., refugees, women, ethnic groups) and from the timelines of politicians and activists. This initial collection yielded over 57,000 tweets. They then manually filtered these tweets to ensure they were in the Levantine dialect, non-Arabic, non-textual, and not duplicates. The remaining tweets were normalized by removing common Twitter artifacts like mentions, hashtags, emojis, and digits.

Crucially, the annotation process involved three native Levantine speakers with higher education. They were provided with detailed guidelines to distinguish between "Normal," "Abusive," and "Hate" speech. The guidelines emphasized that hate speech specifically targets groups based on characteristics like race, gender, or religion, and involves demeaning or dehumanizing language, distinguishing it from general abusive language. To ensure reliability, the authors employed rigorous annotation evaluation metrics, including observed agreement, pairwise percent agreement, Cohen's Kappa, and Krippendorff's alpha.

Datasets

The primary contribution is the introduction of the L-HSAB Twitter dataset. This dataset comprises 5,846 tweets labeled into three categories: Normal, Abusive, and Hate. The dataset is intended to be publicly available. The authors also reviewed existing Arabic hate/abusive speech datasets (e.g., from Alakrota et al., Mubarak et al., Albadi et al., Al-Ajlan and Ykhlef) to contextualize their work and highlight the limitations of prior efforts.

Key Insights and Ideas

  • Dialectal Nuance is Critical: The paper underscores that simply translating or adapting models from standard Arabic or other languages is insufficient for dialects like Levantine, which have unique informalities and cultural contexts.
  • Distinguishing Hate from Abuse is Hard: The authors acknowledge that differentiating hate speech from general abusive language is inherently difficult and prone to annotator bias, often requiring deep contextual understanding.
  • Targeting Specific Groups is Key to Hate Speech: The annotation guidelines highlight that hate speech's defining characteristic is its targeting of specific groups, often based on identity markers, which differentiates it from broader insults or profanity.
  • Annotation Evaluation is Paramount: The paper emphasizes the importance of robust inter-annotator agreement metrics (like Cohen's Kappa and Krippendorff's alpha) to ensure the reliability and quality of any benchmark dataset.
  • Entity Mentions Can Be Misleading: The study found that entity names (e.g., country names) can appear in both normal and hateful contexts, making simple keyword matching unreliable for hate speech detection.

Key Results

  • The L-HSAB dataset contains 5,846 tweets categorized as Normal, Abusive, or Hate.
  • Annotation evaluation showed high agreement among annotators, with Krippendorff's alpha at 76.5%, indicating a reliable dataset, particularly for the minority categories (Abusive and Hate).
  • When tested with machine learning classifiers (SVM and NB), the NB classifier outperformed SVM in both binary (Abusive/Normal) and multi-class (Abusive/Hate/Normal) classification tasks, achieving up to 90.5% precision and 89.6% F1-score for binary classification.

Limitations and Future Work

  • Annotation Agreement on Hate Speech: While overall agreement was high, the paper notes that conflicts arose more frequently when identifying hate speech, suggesting that annotator background knowledge and personal biases play a significant role.
  • Levantine Dialect Specificity: The dataset focuses on the Levantine dialect. The authors suggest that future work should involve building similar datasets for other underrepresented Arabic dialects, such as Tunisian and Gulf.
  • Future Datasets: The paper explicitly calls for the creation of publicly available HS and abusive datasets for other underrepresented Arabic dialects.

Summary updated

BibTeX
@inproceedings{mulki2019l,
  author = {Hala Mulki and Hatem Haddad and Chedi Bechikh Ali and Halima Alshabani},
  title = {{L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure