TL;DR
This paper introduces the novel task of hateful event detection—identifying and classifying real-world hate incidents from textual reports rather than just detecting toxic language online. Evaluating state-of-the-art LLMs on antisemitic incident datasets, the authors find that while models show promise, optimizing their performance requires context-sensitive prompting: explicit definitions work best for rhetoric-oriented harms, whereas in-context examples are crucial for action-oriented events.
What Problem Does This Paper Address?
Prior computational work on tracking hate has overwhelmingly focused on hate speech detection, typically analyzing toxic language in social media posts. However, this approach fails to capture real-world, offline harms like physical violence, vandalism, or institutional discrimination, which are often reported in objective, non-toxic language by journalists or civil society groups. Standard toxicity tools (like Google's Perspective API) routinely miss these reports because the text describing the event isn't inherently abusive.
To bridge this gap, the authors formalize the task of hateful event detection. They focus on antisemitism as a complex, historically grounded case study where definitions are heavily contested and manifestations range from coded tropes to physical assaults.
Methods
The authors systematically evaluate two generative models—OpenAI’s closed-source GPT-4o and Meta’s open-weight Llama-3.2-3B-Instruct—using zero-shot and few-shot prompting strategies. Rather than fine-tuning, they focus on prompting to reflect the practical constraints of non-profit and civil society practitioners.
They test five distinct prompt variations to understand how different types of information affect model reasoning:
- NoCtx (Baseline): Only the incident description, location, and time are provided.
- Assumed: The model is explicitly told to assume the incident is antisemitic, bypassing the binary classification step to focus purely on categorization.
- Def: Includes a general definition of antisemitism alongside brief explanations of each fine-grained label.
- Assumed-Def: Combines the binary assumption with explicit label definitions.
- Assumed-IcE: Combines the binary assumption with one randomly selected, manually labeled in-context example per category (few-shot learning).
Datasets
The study utilizes four distinct datasets to evaluate model performance and generalizability:
- AMCHA: 4,410 expert-annotated reports of antisemitic incidents on US college campuses, featuring a hierarchical taxonomy of two coarse-grained categories and nine fine-grained types.
- ADL-HEAT: 4,522 incident reports from the Anti-Defamation League, categorized into Harassment, Vandalism, and Assault.
- Synthetic (New): A contrast set of 12,402 LLM-generated descriptions of benign, positive Jewish or Israeli cultural events (e.g., a Passover seder or folk dance class) used to rigorously test the models' false-positive rates.
- Campus-News (New): A real-world evaluation set consisting of 5,275 scraped articles from five prominent university newspapers, with a subset of 225 articles consensus-coded by human experts.

Key Insights and Ideas
- The Prompting Dichotomy (Rhetoric vs. Action): The most striking finding is that prompt engineering is not one-size-fits-all. For rhetoric-oriented categories (like historical tropes or genocidal language), providing explicit definitions yields the highest performance gains. For action-oriented categories (like physical assault or property destruction), providing in-context examples is far more effective.
- Stable Narratives vs. Local Context: LLMs perform surprisingly well on abstract, historically grounded categories (e.g., Historical and Genocidal expressions) when given definitions. The authors hypothesize that these draw on stable, globally recognized narratives well-represented in pre-training data. Conversely, models struggle with Bullying and Discrimination, which rely heavily on localized, interpersonal, and situational context.
- Taxonomy Artifacts vs. Model Errors: Error analysis revealed that some "failures" are actually due to rigid annotation guidelines. For instance, AMCHA mandates that if an incident contains both expression and targeting, it must be labeled solely as Targeting. LLMs frequently flagged both, resulting in technical false positives despite being conceptually accurate.
Key Results
- Binary Detection & False Positives: Both models performed well at binary classification. Notably, GPT-4o achieved a 0% false-positive rate on the Synthetic contrast dataset, whereas Llama-3.2-3B misclassified up to 13.36% of benign cultural events as antisemitic when prompted with definitions.
- Fine-Grained Performance: GPT-4o significantly outperformed Llama-3.2-3B across all fine-grained tasks.
- Prompting Impact: On the AMCHA dataset, adding definitions boosted GPT-4o's weighted F1 score for rhetoric types from 56% to 67%. For action types, adding in-context examples boosted the weighted F1 score from 56% to 66%.
- Real-World Application: When deployed on the open-ended Campus-News dataset, GPT-4o acted as an effective initial filter, flagging 364 potential incidents out of 5,275 articles. However, it exhibited high recall but low precision, confirming its utility as a screening tool to assist—rather than replace—human analysts.
Limitations and Future Work
- Selection and Reporting Bias: The datasets rely entirely on incidents that were formally reported or picked up by the media, which represents only a fraction of actual occurrences.
- Geographic and Linguistic Constraints: The study is strictly limited to English-language, U.S.-focused contexts.
- Lack of Fine-Tuning: The authors only explored prompting strategies. Future work should compare these results against task-specific fine-tuning, which might offer more robust performance at the cost of higher computational overhead.
- Rigid Taxonomies: Current models struggle with mutually exclusive hierarchical categories when real-world events inherently blur those boundaries.
Summary updated