TL;DR
This paper introduces HATEMOJICHECK, a novel test suite designed to evaluate how well AI models detect hate speech that uses emojis. They found existing models struggle significantly with emoji-based hate. To address this, they created HATEMOJIBUILD, an adversarially generated dataset, and trained models that show substantial improvements in detecting emoji-based hate without sacrificing performance on text-only hate.
What Problem Does This Paper Address?
Online hate speech is a persistent problem, and automated detection systems are crucial for moderation. However, these systems often struggle with the nuances of human language, especially when emojis are involved. Emojis can be used to convey complex emotions, intentions, and even to express hate in subtle or coded ways, making them a significant challenge for current AI models. Existing hate speech detection benchmarks often overlook or inadequately address this emoji-based hate, leaving a critical gap in our ability to effectively moderate online content. This paper aims to bridge that gap by providing tools and methods to better understand and combat emoji-based hate.
Methods
The authors employed a two-pronged approach:
- HATEMOJICHECK Test Suite: They developed a functional testing framework, inspired by software engineering practices, to systematically evaluate model performance on specific types of emoji-based hate. This involved defining seven distinct "functionalities" (e.g., "Verb Swap," "Identity Swap," "Descriptor Swap," "Append") that represent common ways emojis are used in hateful contexts. For each functionality, they created test cases and "perturbations" – minimal edits to the original test cases designed to probe model vulnerabilities.
- HATEMOJIBUILD Dataset and Adversarial Training: To address the weaknesses identified by HATEMOJICHECK, they implemented a human-and-model-in-the-loop (HML) training scheme. This involved multiple rounds where annotators were tasked with creating examples that a current "target model" would misclassify. These adversarial examples, focusing on emoji-based hate, were used to retrain and improve the target model iteratively. This process yielded the HATEMOJIBUILD dataset, which includes both hateful and non-hateful examples, with labels for hate type and target.
Datasets
- HATEMOJICHECK: A newly introduced benchmark dataset consisting of 3,930 short-form English statements designed to test emoji-based hate detection. It includes 2,126 original test cases and 1,804 perturbations across seven functionalities and six protected identities.
- HATEMOJIBUILD: A newly introduced dataset of 5,912 adversarially generated examples, created through a human-and-model-in-the-loop process. This dataset is balanced between hate and non-hate examples and includes labels for hate type and target.
- Existing Datasets: The paper also references and uses data from previous work, such as Vidgen et al. (2021b) for text-only hate speech data, and mentions using datasets like Davidson et al. (2017) and Founta et al. (2018) for baseline model evaluation.
Key Insights and Ideas
- Emojis are a significant blind spot for hate speech models: The study clearly demonstrates that models trained primarily on text struggle immensely when emojis are introduced, even when the underlying sentiment is similar to text-only hate.
- Adversarial data generation is crucial for robustness: The HML approach, where annotators actively try to fool the model, is highly effective in generating challenging examples that push model performance beyond simple pattern matching.
- Targeted evaluation is key: The HATEMOJICHECK framework, by breaking down emoji-based hate into specific functionalities, reveals why models fail, not just that they fail. This granular insight is vital for targeted improvement.
- Minimal adversarial data can yield significant gains: The paper shows that even a relatively small number of adversarially generated emoji-based hate examples (around 2,000) can lead to substantial improvements in model performance.
Key Results
- Model Weaknesses: Existing commercial (Google Jigsaw's Perspective API) and academic models showed significant weaknesses on HATEMOJICHECK, particularly in identifying hate where identity terms were replaced by emojis (F2 and F4 functionalities).
- HATEMOJIBUILD Performance: Models trained on HATEMOJIBUILD (R6-T, R7-T, R8-T) achieved significantly higher accuracy on HATEMOJICHECK (0.87-0.88) compared to baseline models like P-IA (0.69) and R5-T (0.78).
- Performance Gains: The most substantial performance gain was observed between the R5-T and R6-T models, with an F1-score increase of 0.28 on the emoji test set after training on just 2,000 emoji-containing examples.
- No Trade-off: The newly trained models maintained strong performance on text-only hate speech datasets, indicating that improving emoji detection did not come at the cost of text-based hate detection.
- Fairness: The adversarially trained models (specifically R8-T) showed more balanced subgroup fairness metrics compared to baseline models, suggesting improved robustness across different identity groups.
Limitations and Future Work
- Limitations:
- HATEMOJICHECK is limited to English and does not cover intersectional identities.
- The test cases are synthetically generated and relatively simple, offering negative predictive power rather than comprehensive coverage of real-world hate speech.
- The set of emojis and identities is not exhaustive.
- Future Work:
- Expand HATEMOJICHECK to include more languages and intersectional identities.
- Evaluate against more complex and diverse forms of emoji-based hate.
- Investigate performance differences in more detail using the provided labels for hate type and target.
- Adapt the methodology for other languages and emerging forms of online harm.
Summary updated