Abstract
To protect users from massive hateful content, existing works have focused on automated hate speech detection. However, a critical question remains: do these automated detectors conform to specific social media content policies? Platform policies serve as a contract for moderation, yet existing datasets often lack the granularity to test this conformity. This work introduces HateModerate, a dataset designed to test automated content moderators against specific policies. Through a six-step annotation process involving 28 human annotators and GPT models, we mapped 7.6k examples to Facebook’s 41 hate speech policies. Our evaluation reveals that state-of-the-art detectors frequently fail to uphold these policies. However, we demonstrate that augmenting training data with HateModerate significantly improves policy conformity without compromising general performance.
Introduction
In the current social media landscape, content moderation policies act as a "contract" between platforms and their users. For instance, Facebook’s guidelines are highly specific, encompassing 41 distinct rules that range from dehumanizing metaphors to economic exclusion. Despite the proliferation of automated hate speech detectors, most are trained on general datasets that do not account for these fine-grained rules. This lack of "policy conformity" undermines the transparency and accountability of automated moderation. A model might successfully identify a generic slur but fail to recognize a specific policy violation, such as mocking a victim of a hate crime, or conversely, over-correct by flagging non-hateful speech that simply mentions a protected group.
The HateModerate Pipeline
To bridge this gap, we developed a rigorous six-step annotation pipeline to create HateModerate, combining human expertise with Large Language Model (LLM) capabilities:
- Match: Human annotators searched existing datasets (such as DynaHate, HateCheck, and Ethos) for examples matching Facebook’s 41 policies.
- Augment (Hate): For policies with insufficient real-world examples, GPT-3 was utilized to generate synthetic hateful sentences.
- Verify (Hate): Human annotators verified that these synthetic examples accurately violated the intended policy.
- Retrieve: Using dense retrieval and OpenAI embeddings, the team identified "difficult" non-hateful examples from existing benchmarks that were semantically similar to hateful ones.
- Augment (Non-Hate): GPT-3 generated non-hateful counter-examples to provide contrast (e.g., comparing "Women should be promoted" against "Women are objects").
- Verify (Non-Hate): A final verification step involving both humans and GPT-4 ensured that non-hateful samples were correctly labeled and contextually relevant.
Key Insights and Model Evaluation
Our testing of industry leaders (Google’s Perspective API, OpenAI’s Moderation API) and open-source models (RoBERTa) against HateModerate yielded several key insights:
- Severity Bias: Models are significantly more effective at catching "Tier 1" violations (e.g., direct calls for violence) than "Tier 2 or 3" violations (e.g., stereotyping or contempt). The more subtle the policy violation, the more likely the model is to fail.
- The Non-Hate Struggle: Models frequently fail on "difficult" non-hateful content, often flagging sentences advocating for protected groups simply because they contain keywords associated with those groups.
- OpenAI Performance: Among off-the-shelf solutions, OpenAI’s Moderation API demonstrated the highest conformity (average failure rate of 0.29), though it still struggled with non-hateful nuances.
Improving Policy Conformity
The most significant finding of this research is that policy conformity is trainable. By fine-tuning a RoBERTa model with the HateModerate dataset (+HM), we reduced the overall failure rate on the HateModerate test set from 0.423 to 0.275. Crucially, this fine-tuned model maintained or improved its performance on independent datasets like HateCheck, proving that policy-specific training does not lead to over-fitting or degradation of general detection capabilities.
Conclusion
HateModerate provides a necessary benchmark for evaluating how well automated systems follow the actual rules of the platforms they moderate. While current state-of-the-art models show substantial failures in policy conformity, our results indicate that targeted data augmentation can significantly close this gap, leading to more transparent and reliable automated content moderation.
Summary updated