Testing Hateful Speeches against Policies

Jiangrui Zheng, Xueqing Liu, Girish Budhrani, Wei Yang, Ravishka Rathnasuriya

Abstract

To protect users from massive hateful content, existing works have focused on automated hate speech detection. However, a critical question remains: do these automated detectors conform to specific social media content policies? Platform policies serve as a contract for moderation, yet existing datasets often lack the granularity to test this conformity. This work introduces HateModerate, a dataset designed to test automated content moderators against specific policies. Through a six-step annotation process involving 28 human annotators and GPT models, we mapped 7.6k examples to Facebook’s 41 hate speech policies. Our evaluation reveals that state-of-the-art detectors frequently fail to uphold these policies. However, we demonstrate that augmenting training data with HateModerate significantly improves policy conformity without compromising general performance.

Introduction

In the current social media landscape, content moderation policies act as a "contract" between platforms and their users. For instance, Facebook’s guidelines are highly specific, encompassing 41 distinct rules that range from dehumanizing metaphors to economic exclusion. Despite the proliferation of automated hate speech detectors, most are trained on general datasets that do not account for these fine-grained rules. This lack of "policy conformity" undermines the transparency and accountability of automated moderation. A model might successfully identify a generic slur but fail to recognize a specific policy violation, such as mocking a victim of a hate crime, or conversely, over-correct by flagging non-hateful speech that simply mentions a protected group.

The HateModerate Pipeline

To bridge this gap, we developed a rigorous six-step annotation pipeline to create HateModerate, combining human expertise with Large Language Model (LLM) capabilities:

  1. Match: Human annotators searched existing datasets (such as DynaHate, HateCheck, and Ethos) for examples matching Facebook’s 41 policies.
  2. Augment (Hate): For policies with insufficient real-world examples, GPT-3 was utilized to generate synthetic hateful sentences.
  3. Verify (Hate): Human annotators verified that these synthetic examples accurately violated the intended policy.
  4. Retrieve: Using dense retrieval and OpenAI embeddings, the team identified "difficult" non-hateful examples from existing benchmarks that were semantically similar to hateful ones.
  5. Augment (Non-Hate): GPT-3 generated non-hateful counter-examples to provide contrast (e.g., comparing "Women should be promoted" against "Women are objects").
  6. Verify (Non-Hate): A final verification step involving both humans and GPT-4 ensured that non-hateful samples were correctly labeled and contextually relevant.
The initial 'Match' step involves human annotators identifying policy-violating examples from existing datasets to establish a baseline.
The 'Verify' step ensures that both hateful and non-hateful examples generated or retrieved are accurately labeled according to specific policy criteria.

Key Insights and Model Evaluation

Our testing of industry leaders (Google’s Perspective API, OpenAI’s Moderation API) and open-source models (RoBERTa) against HateModerate yielded several key insights:

  • Severity Bias: Models are significantly more effective at catching "Tier 1" violations (e.g., direct calls for violence) than "Tier 2 or 3" violations (e.g., stereotyping or contempt). The more subtle the policy violation, the more likely the model is to fail.
  • The Non-Hate Struggle: Models frequently fail on "difficult" non-hateful content, often flagging sentences advocating for protected groups simply because they contain keywords associated with those groups.
  • OpenAI Performance: Among off-the-shelf solutions, OpenAI’s Moderation API demonstrated the highest conformity (average failure rate of 0.29), though it still struggled with non-hateful nuances.

Improving Policy Conformity

The most significant finding of this research is that policy conformity is trainable. By fine-tuning a RoBERTa model with the HateModerate dataset (+HM), we reduced the overall failure rate on the HateModerate test set from 0.423 to 0.275. Crucially, this fine-tuned model maintained or improved its performance on independent datasets like HateCheck, proving that policy-specific training does not lead to over-fitting or degradation of general detection capabilities.

Conclusion

HateModerate provides a necessary benchmark for evaluating how well automated systems follow the actual rules of the platforms they moderate. While current state-of-the-art models show substantial failures in policy conformity, our results indicate that targeted data augmentation can significantly close this gap, leading to more transparent and reliable automated content moderation.

Summary updated

Abstract

To protect users from massive hateful content, existing works studied automated hate speech detection. Despite the existing efforts, one question remains: do automated hate speech detectors conform to social media content policies? A platform's content policies are a checklist of content moderated by the social media platform. Because content moderation rules are often uniquely defined, existing hate speech datasets cannot directly answer this question. This work seeks to answer this question by creating HateModerate, a dataset for testing the behaviors of automated content moderators against content policies. First, we engage 28 annotators and GPT in a six-step annotation process, resulting in a list of hateful and non-hateful test suites matching each of Facebook's 41 hate speech policies. Second, we test the performance of state-of-the-art hate speech detectors against HateModerate, revealing substantial failures these models have in their conformity to the policies. Third, using HateModerate, we augment the training data of a top-downloaded hate detector on HuggingFace. We observe significant improvement in the models' conformity to content policies while having comparable scores on the original test data. Our dataset and code can be found in the attachment.
BibTeX
@inproceedings{zheng2024testing,
  author = {Jiangrui Zheng and Xueqing Liu and Girish Budhrani and Wei Yang and Ravishka Rathnasuriya},
  title = {{Testing Hateful Speeches against Policies}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ZqhfGFpPNJ},
}
Figure