Counter-TWIT: An Italian Corpus for Online Counterspeech in Ecological Contexts

Pierpaolo Goffredo, Valerio Basile, Bianca Cepollaro, Viviana PattiNAACL 2022

TL;DR

This paper introduces Counter-TWIT, a novel Italian Twitter corpus annotated for toxic speech and counterspeech within conversational threads. They demonstrate that analyzing counterspeech in its natural "ecological" context, rather than in isolation, significantly improves automated detection models, highlighting the importance of conversational context for understanding online discourse.

What Problem Does This Paper Address?

Online platforms are rife with toxic speech, which can inflict psychological harm and incite bigotry. While social media platforms employ moderation, these measures often target only the most explicit forms, leaving subtle, implicit toxicity to persist. The paper addresses the challenge of how to effectively counter this implicit toxic speech. Specifically, it explores "counterspeech," a strategy of intervening to challenge harmful content. While theoretical work has explored counterspeech in idealized conversational models, its practical application and automated detection in real-world online environments, particularly in Italian, remain underexplored. The authors aim to bridge this gap by creating a resource and conducting experiments to understand and detect counterspeech in its natural online habitat.

Methods

The core of this work is the creation and annotation of the Counter-TWIT corpus. They collected tweets and their replies from Italian Twitter, focusing on accounts that actively call out bigotry. This "ecological" approach aims to capture naturally occurring counterspeech.

The annotation process involved two layers:

  1. Binary Classification: Tweets/replies were labeled as Toxic Speech, Counterspeech, or Support to Counterspeech.
  2. Counterspeech Typology: For identified counterspeech, annotators further classified it into categories like Explicitation, Hostility, Irony/Humor, and Alternative.

They developed a custom annotation platform and used expert annotators trained in linguistics and philosophy. To evaluate their corpus and explore automated detection, they employed a BERT-based neural language model (AlBERTO) pre-trained on Italian tweets. Experiments were conducted using both isolated tweets/replies and concatenated conversational threads (reply + original tweet) to assess the impact of context.

Datasets

The primary dataset introduced is Counter-TWIT, an Italian Twitter corpus. It consists of 624 messages (344 root tweets and 280 replies) annotated for toxic speech, counterspeech, and support for counterspeech. A second layer of annotation categorizes counterspeech into Explicitation, Hostility, Irony/Humor, and Alternative. This corpus is unique in its focus on "ecological" conversational threads and its Italian language focus.

Key Insights and Ideas

  • Context is King for Counterspeech: The most significant insight is that analyzing counterspeech within its conversational context (i.e., including the original tweet it replies to) dramatically improves the performance of automated detection models, particularly for recall. This validates the intuition that counterspeech is inherently a "second-turn" intervention that relies on the preceding discourse.
  • Implicit Toxicity is Tricky: The paper highlights that implicit toxic content is particularly insidious because it can evade detection and be accepted without critical scrutiny. Counterspeech, especially through explicitation, aims to bring this implicit harm to light.
  • Counterspeech is Sparse and Diverse: Collecting counterspeech in the wild is challenging due to its rarity. The corpus reveals that counterspeech can manifest in various forms (hostility, irony, explicitation, alternatives), making robust detection a complex task.
  • Hostility and Irony are Hard to Model: The automated models struggled most with detecting ironic and hostile counterspeech, suggesting these forms are more ambiguous or cognitively demanding to process computationally compared to explicitation or alternative suggestions.

Key Results

  • Improved Detection with Context: Including conversational context (root tweet + reply) in training the AlBERTo model led to significant improvements, especially in recall for counterspeech detection (a 65% increase).
  • Model Performance: The models achieved reasonable performance in binary classifications (toxic vs. non-toxic, counterspeech vs. non-counterspeech, support vs. non-support). For example, the F1-score for counterspeech detection using reply text alone was 0.584, improving to 0.742 when context was included.
  • Annotation Challenges: Inter-annotator agreement (Krippendorff's alpha) was generally moderate, with lower agreement on replies compared to root tweets. The "Explicitation" label proved particularly difficult to annotate consistently, indicating its nuanced nature.
  • False Positive Analysis: The model tended to confuse hostile and ironic counterspeech more than explicit or alternative forms, with FP ratios around 60-70% for irony and hostility.

Limitations and Future Work

  • Annotation Subjectivity: The authors acknowledge that the subjective nature of identifying toxic speech and counterspeech, especially implicit forms, contributes to annotation disagreement.
  • Limited Corpus Size: While novel, the Counter-TWIT corpus is relatively small (624 messages), which can impact the robustness of automated models.
  • Future Directions: The authors plan to expand the corpus and conduct further qualitative analyses to understand how language models interpret different counterspeech categories. They also aim to develop models for automatically generating counterspeech to combat hate speech, a crucial next step in addressing online toxicity.

Summary updated

BibTeX
@inproceedings{goffredo2022counter,
  author = {Pierpaolo Goffredo and Valerio Basile and Bianca Cepollaro and Viviana Patti},
  title = {{Counter-TWIT: An Italian Corpus for Online Counterspeech in Ecological Contexts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure