TL;DR
This paper introduces DALC v1.0, the first comprehensive, manually annotated Dutch dataset for detecting abusive language on Twitter. It addresses the critical need for language-specific resources to combat online toxicity, employing a novel multi-layered annotation scheme and a hybrid data collection strategy to minimize bias.
What Problem Does This Paper Address?
The proliferation of abusive language on social media platforms poses a significant challenge for content moderation. While automatic detection systems are crucial, their effectiveness hinges on the availability of high-quality, language-specific datasets. Existing resources for Dutch are scarce, and many datasets suffer from biases introduced by their collection methods or limited scope. This paper aims to fill this gap by creating DALC v1.0, a dataset designed to be robust, representative, and compatible with previous work, enabling better detection of abusive language in Dutch.
Methods
The authors employed a multi-pronged approach to data collection, combining three strategies to mitigate bias:
- Keyword Extraction: This method leveraged controversial posts from the Dutch Reddit community "r/thenetherlands" to identify relevant keywords. These keywords were then used to search for tweets in corresponding time periods on Twitter. This approach aims to avoid developer bias in keyword selection.
- Geolocation: Inspired by research linking online hate speech to economically depressed areas, this method focused on collecting tweets from geographical regions in the Netherlands with high unemployment rates. This was achieved by using city names within those provinces as search terms.
- Seed Users: This strategy involved identifying users who frequently post abusive content. An ad-hoc list of profanities and slurs was compiled, and users who posted messages containing these terms were identified. A subset of these users was then selected as "seed users" to collect their tweets.
For annotation, DALC v1.0 utilizes a multi-layered scheme. The explicitness layer distinguishes between "Explicit" (containing profanities/slurs), "Implicit" (potentially abusive but without explicit cues), and "Not abusive" messages. The target layer categorizes messages based on whether they target an "Individual" (IND), a "Group" (GRP), or "Other" (OTH) entities.
Datasets
The primary contribution is the Dutch Abusive Language Corpus (DALC) v1.0. This dataset comprises 8,156 manually annotated Dutch tweets. The authors also conducted baseline experiments using this dataset, comparing a dictionary-based approach, a Linear SVM, and a Dutch transformer-based language model (BERTje).
Key Insights and Ideas
- Hybrid Data Collection is Key: Combining keyword extraction, geolocation, and seed users proved more effective than relying on a single method, leading to a more diverse and less biased dataset.
- Implicit Abuse is Tricky: Distinguishing between "Implicit" abusive messages and "Not abusive" messages is particularly challenging, even for advanced models like BERTje. This highlights the subtlety of online abuse.
- Target Classification is Difficult: While models can identify abusive language, accurately classifying the target (individual, group, or other) remains a significant challenge, with macro-F1 scores below 0.50.
- Data Overlap Matters: The authors deliberately avoided overlap in time periods and users between their training and testing sets to create a more challenging and realistic evaluation scenario. This approach, while increasing difficulty, validates the robustness of their data curation.
Key Results
- Explicitness Classification: The BERTje model achieved the best performance with a macro F1 score of 0.561 for the fine-grained explicitness classification (EXP, IMP, NOT). For binary classification (ABU vs. NOT), BERTje achieved a macro F1 of 0.748.
- Target Classification: The BERTje model achieved a macro F1 score of 0.498 for target classification (IND, GRP, OTH).
- Abusive Message Proportion: Approximately one-third of the messages in DALC v1.0 are labeled as abusive.
- Message Length: Abusive messages are, on average, longer than non-abusive ones (27.58 words vs. 22.77 words).
Limitations and Future Work
The authors acknowledge that the task of detecting abusive language, especially implicitly abusive messages and their targets, remains challenging. They note that the increased granularity of the explicitness classes negatively impacts the performance of simpler models like SVMs.
Future work will focus on an in-depth investigation of errors to identify easy and complex cases. The authors also plan to explore more advanced techniques for target classification and further refine the dataset and annotation guidelines. They also highlight the potential for dual use of the dataset and models, emphasizing the need for responsible development and deployment.
Summary updated