TL;DR
This paper introduces HATECAT-TR, the first span-annotated dataset for hate speech detection and categorization in Turkish. By combining GPT-40's automated annotation with human expert review, they've created a valuable resource for understanding and combating nuanced hate speech in Turkish social media, outperforming existing models on this challenging task.
What Problem Does This Paper Address?
Effectively moderating online content, especially hate speech, is a significant challenge. While many systems can flag entire posts as hateful, they often lack the granularity to pinpoint the specific problematic phrases. This makes it difficult for human moderators to quickly assess the severity and nature of the hate speech, leading to less efficient and potentially inconsistent moderation. Furthermore, existing Turkish hate speech datasets are limited, particularly in their ability to perform fine-grained span detection and categorization, which is crucial for understanding the subtle ways hate speech manifests. This paper aims to fill this gap by providing a comprehensive dataset and baseline models for Turkish hate speech span detection and categorization.
Methods
The core of this work is the creation of the HATECAT-TR dataset. They employed a semi-automated approach:
- Automated Span Annotation: GPT-40 was used to generate initial hate speech spans and their categories within tweets. A structured system message guided GPT-40, and each tweet was pre-assigned a target group.
- Human Expert Review: The GPT-40 generated spans underwent a rigorous two-stage manual review. This involved annotators deleting, creating, or modifying spans to ensure accuracy and assigning them to one of five predefined hate speech categories. Disagreements between annotators were resolved by a third reviewer.
- Task Framing: Hate speech span detection and categorization were framed as both binary and multi-class token classification tasks.
Datasets
- HATECAT-TR: A newly introduced dataset containing 4465 hateful spans across 2981 Turkish tweets, targeting eight minority groups. Each span is categorized into one of five discourse types.
- Prior Datasets: The paper mentions that part of their initial dataset was previously used in the HSD2Lang Hate Speech Detection Challenge.
Key Insights and Ideas
- Implicit Hate is Hard: The study highlights that detecting and categorizing implicit hate speech, especially when spans are subtle and highly contextual, is significantly more challenging for both models and human annotators.
- Turkish-Specific Models Shine: Turkish-specific language models (BERTurk and TurkishBERTweet) outperformed multilingual models (LLaMA, Gemma) on the hate speech tasks, underscoring the importance of language-specific pretraining for nuanced understanding.
- Explicit Threats are Easier: Models performed best on the "Threat of Enmity/War/Attack/Murder/Harm" category, suggesting that direct and explicit forms of hate speech are more readily identifiable.
- Categorization is Tougher: Even for human annotators, agreeing on specific hate speech categories proved difficult, indicating the subjective and nuanced nature of these classifications.
- Semi-Automated Annotation Works: Combining LLM generation with human review significantly reduced the annotation workload while ensuring high-quality, reliable data.
Key Results
- Dataset Size: HATECAT-TR contains 4465 hateful spans from 2981 Turkish tweets.
- Model Performance: BERTurk achieved the highest F1-scores for both span detection and categorization tasks among the evaluated models.
- Category Performance: Models performed best on "Threat of Enmity; War; Attack; Murder; or Harm" (BERTurk F1 ~0.44) and struggled with "Exclusive/Discriminatory Discourse" and "Exaggeration; Generalization; Attribution; Distortion."
- Inter-Annotator Agreement: While overall detection agreement was good (F1=0.77), category agreement was lower (F1=0.50), with "Symbolization" being the most challenging category (F1=0.29).
Limitations and Future Work
- Annotator Subjectivity: Despite the rigorous process, annotator subjectivity and ambiguity in classifying implicit hate speech remain challenges.
- Dataset Bias: The dataset is limited to Turkish tweets, potentially introducing platform-specific biases. Some hate speech categories are also underrepresented.
- Model vs. Human Performance: Models don't always align with human annotators, suggesting they might rely on different cues.
- Turkish Morphology: The agglutinative nature of Turkish presents challenges for span-level detection, potentially impacting subword tokenization and alignment.
- Future Work: The authors suggest expanding dataset diversity, improving contextual understanding, and adapting to the evolving nature of online hate speech. Further research into improving model performance on implicit hate speech and addressing the challenges of Turkish morphology is also recommended.
Summary updated