TL;DR
This paper introduces DALC v2.0, a significantly expanded and refined Dutch corpus for detecting offensive and abusive language on social media. By introducing a new annotation layer for explicitness and refining target identification, DALC v2.0 offers a more nuanced resource for training and evaluating NLP models, achieving state-of-the-art results with Transformer-based models.
What Problem Does This Paper Address?
Social media platforms are rife with socially unacceptable language, posing significant challenges for moderation and research. While automatic systems are crucial for flagging such content, they heavily rely on language-specific datasets. Existing Dutch resources were limited, prompting the need for a more comprehensive and detailed corpus. This paper addresses this gap by presenting DALC v2.0, which not only expands the existing Dutch Abusive Language Corpus (DALC) but also introduces a more granular annotation scheme to better capture the nuances of offensive language.
Methods
The authors built upon the existing DALC v1.0 corpus, expanding it to 11,292 messages. A key innovation is the introduction of a multi-layer annotation scheme. This scheme includes:
- Explicitness: Distinguishing between explicitly offensive language (e.g., profanity), implicitly offensive language (subtle or hidden offense), and non-offensive messages.
- Target: Identifying whether the offense is directed at an individual, a group, or another entity (concepts, institutions), and also allowing for messages without a target.
They also refined the definition of abusive language and conducted extensive experiments using various NLP models, including SVM, Bi-LSTM, and Transformer-based models like BERTje and RobBERT, to evaluate their performance on the new annotation dimensions.
Datasets
- DALC v2.0: The primary contribution is this newly annotated corpus of 11,292 Dutch Twitter messages. It extends the previous DALC v1.0 (8,156 messages) by adding over 3,000 new messages and introducing the explicitness annotation layer.
- Existing Dutch Abusive Language Corpus (DALC v1.0): The foundation for DALC v2.0.
Key Insights and Ideas
- Subjectivity is Key: The authors highlight that offensiveness is inherently subjective, leading to lower inter-annotator agreement on the explicitness layer compared to other linguistic tasks. This subjectivity is a crucial factor in understanding model performance and limitations.
- Nuance Matters: The distinction between explicit and implicit offensive language, and the identification of targets, provides a richer understanding of online discourse than simple binary classification.
- Transformer Models Lead: Monolingual Transformer-based language models (RobBERT and BERTje) consistently outperform other architectures, demonstrating their effectiveness in capturing the complexities of offensive language.
- Abusive vs. Offensive Distinction is Challenging: Differentiating between abusive and offensive language is a more complex task for models, suggesting that while systems can identify individual phenomena, distinguishing between them requires more sophisticated approaches.
Key Results
- Best Performance: RobBERT achieved the highest macro-average F1 score of 0.828 for binary classification of offensive language and 0.637 for distinguishing between abusive and offensive messages. BERTje performed best on target identification with a macro-average F1 of 0.579.
- Inter-Annotator Agreement: While agreement was moderate for explicitness (Fleiss' Kappa of 0.430) and slightly lower for target identification (Fleiss' Kappa of 0.402), the authors argue that disagreements can be informative due to the subjective nature of the task.
- Data Distribution: The corpus exhibits an unbalanced distribution of negative and positive examples, which is considered more realistic for social media data.
- Error Analysis: Models tend to rely on superficial linguistic cues and are sensitive to data distribution. A significant portion of errors are attributed to bias in the data, including false positives for offensive language containing identity terms.
Limitations and Future Work
- Model Limitations: Systems heavily rely on surface cues and struggle with deeper comprehension, leading to errors related to irony, sarcasm, and world knowledge. The imbalance in the dataset also impacts model performance, particularly recall for offensive classes.
- Data Limitations: The error analysis revealed potential annotation mistakes in the gold standard data and a bias towards certain identity terms.
- Future Work: The authors plan to explore multilingual approaches, test models on dynamic benchmarks, and further investigate the impact of language versus domain shift in transfer learning. They also emphasize the need for caution when deploying models due to their imperfect performance and the potential for misrepresentation.
Summary updated