Mapping Toxic Comments Across Demographics: A Dataset from German Public Broadcasting

Jan Fillies, Michael Peter Hoffmann, Rebecca Reichel, Roman Salzwedel, Sven Bodemer, Adrian PaschkeEMNLP 2025

TL;DR

This paper introduces the first large-scale German dataset of toxic social media comments annotated with user age demographics, collected from public broadcasting platforms. By analyzing this dataset, the authors reveal significant age- and platform-based differences in toxic language, offering a crucial resource for developing more equitable and age-aware content moderation systems.

What Problem Does This Paper Address?

Understanding online toxicity is crucial, but existing datasets often lack demographic context, particularly age. This makes it difficult to grasp how different age groups communicate online and how toxicity manifests across these demographics. While platforms restrict access to user data, this research leverages a unique partnership with "funk," a German public service content network, to access anonymized, platform-provided age estimates for users. This allows for a more nuanced analysis of toxic speech patterns across age groups and platforms, addressing a significant gap in current research.

Methods

The researchers developed a multi-stage annotation process. First, they curated a dataset of 3,024 human-annotated comments. To scale this effort, they then employed and evaluated four large language models (LLMs) – GPT-3.5, GPT-40, GPT-40-mini, and Llama-3-8B – for toxicity annotation. The best-performing LLM, GPT-40-mini, was fine-tuned and used to annotate an additional 30,024 comments. The annotation scheme was based on "funk's" content policies and included 18 labels across two main classes: target of toxic language and type of language. Statistical analyses, including chi-squared tests, were used to identify significant differences in label distributions across age groups and platforms.

Datasets

The paper introduces two novel datasets:

  • Human-annotated dataset: 3,024 anonymized comments from Instagram, TikTok, and YouTube, annotated by three experienced researchers.
  • LLM-annotated dataset: 30,024 anonymized comments, annotated using a fine-tuned GPT-40-mini model, based on the human annotation schema.

Both datasets are derived from comments on "funk's" social media channels and their associated accounts. The dataset is published via Zenodo and available to verified researchers.

Key Insights and Ideas

  • Age Matters: Toxic speech patterns are not uniform across age groups. Younger users (0-30) tend to use more expressive language and emojis, while middle-aged users (31-35) show higher rates of insults and disinformation. Older users (35+) also exhibit increased disinformation and devaluation, particularly related to gender and sex.
  • Platform Differences are Stark: Instagram emerges as the platform with the highest frequency of toxic content, particularly insults, devaluation, disinformation, and discrimination. YouTube and TikTok show lower toxicity rates, with TikTok exhibiting the least hate speech overall.
  • LLMs as Scalable Annotators: LLMs, particularly GPT-40 and GPT-40-mini, demonstrate high agreement with human annotators, offering a cost-effective and efficient way to scale annotation efforts for large datasets. However, LLM annotations can sometimes differ from human annotations, especially at the platform level, highlighting the need for careful interpretation.
  • Context is King (and Often Missing): The lack of full conversation threads or video context poses a challenge for accurate toxicity classification, potentially leading to misclassifications.

Key Results

  • The human-annotated dataset achieved an average inter-annotator agreement of 88.2% (percentage agreement) and 0.61 (Cohen's kappa).
  • GPT-40 showed the highest agreement with human annotators (0.81 Cohen's kappa), outperforming the average human annotator.
  • GPT-3.5 was the fastest LLM for annotation, while GPT-40-mini was the most cost-effective.
  • The LLM-annotated dataset revealed that "non-toxic" comments constituted 77.02% of the data, a slight decrease from the human-annotated set.
  • "Insults" (6.51%), "Devaluation" (5.5%), and "Disinformation" (2.78%) were the most common toxic labels in the human-annotated dataset.
  • A chi-squared test indicated statistically significant differences in label distributions across age groups (p < 0.05).
  • Instagram showed the highest toxicity frequency (22.53%), followed by YouTube (13.03%) and TikTok (12.43%).

Limitations and Future Work

The authors acknowledge several limitations:

  • Age Data Reliance: The age estimates are derived from platform-level audience data, which may not perfectly reflect individual user demographics and could be influenced by users misrepresenting their age or by topic differences within channels.
  • Platform-Specific Biases: Moderation policies and platform designs can influence the visibility and collection of toxic content, potentially leading to underestimation of certain types of toxicity.
  • LLM Annotation Errors: LLMs can make errors, exhibit biases, and their closed-source nature limits transparency and auditability.
  • Contextual Understanding: The absence of full conversation threads or video context can hinder accurate classification.

Future work should focus on incorporating context-aware toxicity modeling, explicitly addressing moderation dynamics, and exploring the impact of topic differences on age-related toxicity. The dataset is intended to support fairer, data-driven moderation strategies and further research into multilingual and demographically-aware toxicity detection.

Summary updated

BibTeX
@inproceedings{fillies2025mapping,
  author = {Jan Fillies and Michael Peter Hoffmann and Rebecca Reichel and Roman Salzwedel and Sven Bodemer and Adrian Paschke},
  title = {{Mapping Toxic Comments Across Demographics: A Dataset from German Public Broadcasting}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure