Detecting Abusive Albanian

Erida Nurce, Jorgel Keci, Leon DerczynskiarXiv 2021 · arXiv

Abstract

The ever growing usage of social media in the recent years has had a direct impact on the increased presence of hate speech and offensive speech in online platforms. Research on effective detection of such content has mainly focused on English and a few other widespread languages, while the leftover majority fail to have the same work put into them and thus cannot benefit from the steady advancements made in the field. In this paper we present Shaj, an annotated Albanian dataset for hate speech and offensive speech that has been constructed from user-generated content on various social media platforms. Its annotation follows the hierarchical schema introduced in OffensEval. The dataset is tested using three different classification models, the best of which achieves an F1 score of 0.77 for the identification of offensive language, 0.64 F1 score for the automatic categorization of offensive types and lastly, 0.52 F1 score for the offensive language target identification.

Introduction

While hate speech detection has advanced rapidly for high-resource languages, "low-resource" languages like Albanian have been largely ignored. This creates a significant gap in content moderation; as social media usage grows in Albania, so does the presence of homophobic, sexist, and racist content. This paper introduces Shaj (Spoken Hate in the Albanian Jargon), the first publicly available dataset for detecting hate speech and offensive language in Albanian, establishing a vital benchmark for the field.

The Shaj Dataset

The Shaj dataset consists of 11,874 comments scraped from high-engagement public accounts on Instagram and YouTube to capture authentic, informal jargon. The data is naturally skewed, with approximately 87% of comments labeled as "Not Offensive," reflecting the real-world challenge of finding offensive content in a vast stream of data.

Methodology

The researchers employed a three-tier classification approach based on the OffensEval (OLID) schema:

  1. Subtask A: Offensive vs. Not Offensive.
  2. Subtask B: Automatic categorization (Targeted vs. Untargeted insults).
  3. Subtask C: Target identification (Individual vs. Group vs. Other).

To benchmark the dataset, three models were compared: a Naive Bayes baseline, a BiLSTM using Albanian-specific FastText embeddings, and a fine-tuned Multilingual BERT (bert-base-multilingual-uncased). Notably, emoticons were preserved in the text as they are often essential for understanding sarcastic intent.

Experimental Results

BERT consistently outperformed the other models across all subtasks. For the primary task of identifying offensive language (Subtask A), BERT achieved an F1 score of 0.77, significantly higher than the BiLSTM (0.70) and Naive Bayes (0.41) models.

Performance decreased for the more granular tasks: BERT achieved an F1 score of 0.64 for Subtask B and 0.52 for Subtask C. This drop is largely attributed to the smaller volume of "offensive" training data available for these specific categories.

Discussion and Conclusion

The study highlights unique linguistic challenges in Albanian, such as cultural metaphors (e.g., using "pumpkin" to imply stupidity) and sarcasm that standard models often miss. The "Other" category in target identification proved particularly difficult due to its lack of a unified definition compared to "Individual" or "Group" targets.

While BERT's contextual understanding proved superior even in this low-resource setting, future work must address data imbalance and capture more diverse Albanian dialects. Expanding the dataset to include more balanced classes and sophisticated sarcasm modeling remains a priority for improving automated moderation in the Albanian digital space.

Summary updated

Abstract

The ever growing usage of social media in the recent years has had a direct impact on the increased presence of hate speech and offensive speech in online platforms. Research on effective detection of such content has mainly focused on English and a few other widespread languages, while the leftover majority fail to have the same work put into them and thus cannot benefit from the steady advancements made in the field. In this paper we present \textsc{Shaj}, an annotated Albanian dataset for hate speech and offensive speech that has been constructed from user-generated content on various social media platforms. Its annotation follows the hierarchical schema introduced in OffensEval. The dataset is tested using three different classification models, the best of which achieves an F1 score of 0.77 for the identification of offensive language, 0.64 F1 score for the automatic categorization of offensive types and lastly, 0.52 F1 score for the offensive language target identification.
BibTeX
@misc{nurce2021detecting,
  author = {Erida Nurce and Jorgel Keci and Leon Derczynski},
  title = {{Detecting Abusive Albanian}},
  year = {2021},
  eprint = {2107.13592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.13592},
}
Figure