Abstract
To protect users from massive hateful content, existing works studied automated hate speech detection. Despite the existing efforts, one question remains: Do automated hate speech detectors conform to social media content policies? A platform’s content policies are a checklist of content moderated by the social media platform. Because content moderation rules are often uniquely defined, existing hate speech datasets cannot directly answer this question. This work seeks to answer this question by creating HateModerate, a dataset for testing the behaviors of automated content moderators against content policies. First, we engage 28 annotators and GPT in a six-step annotation process, resulting in a list of hateful and non-hateful test suites matching each of Facebook’s 41 hate speech policies. Second, we test the performance of state-of-the-art hate speech detectors against HateModerate, revealing substantial failures these models have in their conformity to the policies. Third, using HateModerate, we augment the training data of a top-downloaded hate detector on HuggingFace. We observe significant improvement in the models’ conformity to content policies while having comparable scores on the original test data. Our dataset and code can be found on https://github.com/stevens-textmining/HateModerate.
Introduction
In the ecosystem of social media, platform policies—such as Facebook’s Community Standards—serve as a formal "contract" between the service provider and its users. Users expect automated moderation systems to enforce these specific, often granular rules consistently. However, research from the Stevens Institute of Technology, UT Dallas, and JP Morgan AI Research identifies a critical gap: most state-of-the-art hate speech detectors are not benchmarked against these specific policies. Consequently, while a model might excel at general toxicity detection, it often fails to adhere to the nuanced "contractual" obligations defined by the platform.
Existing datasets typically categorize content broadly as "hateful" or "not hateful." Real-world moderation, however, is governed by complex, multi-tiered policies. For instance, Facebook distinguishes between "dehumanizing speech" (Tier 1) and "economic exclusion" (Tier 3). Without testing against these specific rules, there is a lack of transparency and accountability in how platforms moderate content.
The HateModerate Dataset
To address this, the researchers developed HateModerate, a novel dataset comprising 7,704 examples (4,796 hateful, 2,908 non-hateful) mapped directly to 41 distinct Facebook hate speech policies. The construction of HateModerate followed a rigorous six-step pipeline involving manual search, LLM augmentation, and human verification.
The pipeline specifically focused on identifying "hard negatives"—non-hateful examples that are semantically similar to hateful ones. This prevents models from relying on simple keyword detection and forces them to understand the context and policy nuances. The dataset draws from a wide range of sources, including DynaHate, HateCheck, and Twitter Hate Speech, to build a robust retrieval pool.
Key Insights and Results
The benchmarking of four major models—Google’s Perspective API, OpenAI’s Moderation API, and two RoBERTa-based models—revealed several critical insights:
- Severity Bias: Models are significantly more effective at identifying "Tier 1" violations (e.g., dehumanization) than "Tier 3" or "Tier 4" violations (e.g., stereotyping or economic exclusion). The "quieter" forms of hate speech are frequently overlooked by general-purpose detectors.
- The "Hard Negative" Problem: Models often misclassify non-hateful content as hateful if it contains "trigger" words. For example, a post attacking a sexist statement (counter-hate) might be flagged because it quotes the original offensive language.
- OpenAI Leads the Pack: Among tested off-the-shelf tools, OpenAI’s Moderation API showed the highest conformity to specific policies, though it still struggled with certain granular rules.
- Policy-Aware Training: The study demonstrated that adding a small amount of policy-specific data from HateModerate to a model's training set significantly improved its rule conformity. For the Cardiff NLP model, the failure rate on the HateModerate test set dropped from 0.423 to 0.275 after fine-tuning.
- Generalization and Bias: Fine-tuning also improved performance on standard benchmarks like HateCheck (reducing failure rates from 0.365 to 0.199) without introducing new demographic biases against protected groups.
Conclusion
HateModerate serves as a vital stress test for the AI systems policing digital town squares. The findings suggest that general-purpose hate speech detection is insufficient for platform-specific moderation. To ensure AI follows the rules, it must be explicitly trained on the granular policies it is meant to enforce. Future work will need to address the evolution of policies and the detection of more implicit "dog whistle" speech across diverse platforms.
Summary updated