Factoring Hate Speech: A New Annotation Framework to Study Hate Speech in Social Media

Gal Ron, Effi Levi, Odelia Oshri, Shaul R. ShenhavACL 2023

TL;DR

This paper introduces a novel annotation framework that breaks down hate speech into five distinct discursive aspects, moving beyond simple classification. They applied this framework to a large corpus of Twitter conversations targeting Jewish individuals, providing a more nuanced understanding of hate speech's multifaceted nature.

What Problem Does This Paper Address?

Identifying and understanding hate speech on social media is a significant challenge. Existing research often struggles with the amorphous and subjective nature of "hate," leading to broad classifications that may miss crucial nuances. This paper tackles this by proposing a more granular approach to analyzing hate speech, specifically focusing on expressions directed at Jewish individuals on Twitter. The goal is to move beyond simply labeling content as "hate" to understanding how hate is expressed.

Methods

The core contribution is a new annotation scheme that factors hate speech into five separate discursive categories: Contempt, Abuse, Call for Anti-Group Action, Prejudice, and Holocaust Denial. This approach aims to make annotation more objective by decomposing the concept of hate into more specific, narrowly defined aspects. The authors then applied this scheme to a sample of 1,050 tweets from a larger corpus they constructed. They also performed a statistical analysis of the annotated data, including inter-category correlations, to understand how these different aspects of hate speech co-occur.

Datasets

The authors constructed a corpus of over 2.9 million Twitter conversations in English, collected via the Twitter API v2. This corpus was filtered using both neutral and racial keywords related to Jewish individuals. They then annotated a sample of 1,050 tweets from this corpus using their newly developed annotation scheme. This annotated sample dataset is a new contribution for studying hate speech.

Key Insights and Ideas

  • Decomposition of Hate: The paper argues that breaking down "hate" into distinct discursive aspects (Contempt, Abuse, etc.) makes the concept more manageable and objective for annotation and analysis.
  • Nuance in Expression: The five categories allow for a more nuanced understanding of how hate is conveyed, distinguishing between demeaning language (Abuse) and prejudiced beliefs (Prejudice), for instance.
  • Context Matters (Implicitly): While their current annotation doesn't fully capture conversational context, the authors acknowledge its importance and plan to address it in future work, recognizing that agreement with a hateful post can itself be hate speech depending on the surrounding conversation.
  • Inter-Category Relationships: The analysis of inter-category correlations reveals that certain aspects of hate speech tend to co-occur, such as Abuse and Call for Anti-Group Action, suggesting common underlying motivations or expression styles.

Key Results

  • The most common category in their annotated sample was Abuse (181 instances out of 1,050 tweets).
  • Call for Anti-Group Action was the second most common, followed by Prejudice, and then Holocaust Denial as the least common.
  • Inter-category correlations were generally low, suggesting that the categories capture distinct aspects. However, Abuse showed moderate correlations with Call for Anti-Group Action (r = 0.2332) and Prejudice (r = 0.1644).
  • A minor correlation was found between Prejudice and Contempt (r = 0.1227), suggesting prejudiced beliefs might serve as a "rationale" for contempt.
  • A minor correlation between Prejudice and Holocaust Denial (r = 0.1388) was attributed to their association with conspiracy theories.

Limitations and Future Work

  • Single Target Group: The current analysis is limited to hate speech directed at Jewish individuals. While the scheme is designed to be general, its applicability to other groups needs further evaluation.
  • Contextual Understanding: The annotation scheme does not fully account for the context of the surrounding conversation, which can be crucial for identifying hate speech (e.g., agreement with a hateful post).
  • Reader Perception: The scheme does not currently capture how hate speech is perceived by readers, which could be inferred from replies and engagement metrics.

Future work includes:

  • Applying the annotation scheme to other target groups.
  • Annotating complete conversations to capture context.
  • Exploring counter-hate speech and its dynamics.
  • Training additional annotators to create a larger, comprehensive annotated dataset.
  • Assembling and annotating a similar corpus for Muslim-related hateful expressions.

Summary updated

BibTeX
@inproceedings{ron2023factoring,
  author = {Gal Ron and Effi Levi and Odelia Oshri and Shaul R. Shenhav},
  title = {{Factoring Hate Speech: A New Annotation Framework to Study Hate Speech in Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure