Evaluating Large Language Models for Antisemitic Incident Classification

Karina Halevy, Julia Mendelsohn, Chan Young Park, Yulia Tsvetkov, Maarten SaparXiv 2026 · arXiv

TL;DR

This paper introduces the novel task of hateful event detection—identifying and classifying real-world hate incidents from textual reports rather than just detecting toxic language online. Evaluating state-of-the-art LLMs on antisemitic incident datasets, the authors find that while models show promise, optimizing their performance requires context-sensitive prompting: explicit definitions work best for rhetoric-oriented harms, whereas in-context examples are crucial for action-oriented events.

What Problem Does This Paper Address?

Prior computational work on tracking hate has overwhelmingly focused on hate speech detection, typically analyzing toxic language in social media posts. However, this approach fails to capture real-world, offline harms like physical violence, vandalism, or institutional discrimination, which are often reported in objective, non-toxic language by journalists or civil society groups. Standard toxicity tools (like Google's Perspective API) routinely miss these reports because the text describing the event isn't inherently abusive.

Figure 1: We introduce the novel task of fine-grained hateful event detection. This task is distinct from hate speech or toxicity detection: an example report describing an antisemitic incident receives low scores from Perspective API, a popular toxicity detection tool at the time of our dataset’s collection, now being sunset. When only prompted with the event description, the LLM GPT-4o does not identify this example incident as antisemitic. Providing explicit definitions of fine-grained types of harm enables the model to correctly label the incident.

To bridge this gap, the authors formalize the task of hateful event detection. They focus on antisemitism as a complex, historically grounded case study where definitions are heavily contested and manifestations range from coded tropes to physical assaults.

Methods

The authors systematically evaluate two generative models—OpenAI’s closed-source GPT-4o and Meta’s open-weight Llama-3.2-3B-Instruct—using zero-shot and few-shot prompting strategies. Rather than fine-tuning, they focus on prompting to reflect the practical constraints of non-profit and civil society practitioners.

They test five distinct prompt variations to understand how different types of information affect model reasoning:

  • NoCtx (Baseline): Only the incident description, location, and time are provided.
  • Assumed: The model is explicitly told to assume the incident is antisemitic, bypassing the binary classification step to focus purely on categorization.
  • Def: Includes a general definition of antisemitism alongside brief explanations of each fine-grained label.
  • Assumed-Def: Combines the binary assumption with explicit label definitions.
  • Assumed-IcE: Combines the binary assumption with one randomly selected, manually labeled in-context example per category (few-shot learning).
Prompt variations evaluated for event classification

Datasets

The study utilizes four distinct datasets to evaluate model performance and generalizability:

  1. AMCHA: 4,410 expert-annotated reports of antisemitic incidents on US college campuses, featuring a hierarchical taxonomy of two coarse-grained categories and nine fine-grained types.
  2. ADL-HEAT: 4,522 incident reports from the Anti-Defamation League, categorized into Harassment, Vandalism, and Assault.
  3. Synthetic (New): A contrast set of 12,402 LLM-generated descriptions of benign, positive Jewish or Israeli cultural events (e.g., a Passover seder or folk dance class) used to rigorously test the models' false-positive rates.
  4. Campus-News (New): A real-world evaluation set consisting of 5,275 scraped articles from five prominent university newspapers, with a subset of 225 articles consensus-coded by human experts.

Label descriptions and distributions for the AMCHA dataset Label descriptions and frequency for the ADL-HEAT dataset

Key Insights and Ideas

  • The Prompting Dichotomy (Rhetoric vs. Action): The most striking finding is that prompt engineering is not one-size-fits-all. For rhetoric-oriented categories (like historical tropes or genocidal language), providing explicit definitions yields the highest performance gains. For action-oriented categories (like physical assault or property destruction), providing in-context examples is far more effective.
  • Stable Narratives vs. Local Context: LLMs perform surprisingly well on abstract, historically grounded categories (e.g., Historical and Genocidal expressions) when given definitions. The authors hypothesize that these draw on stable, globally recognized narratives well-represented in pre-training data. Conversely, models struggle with Bullying and Discrimination, which rely heavily on localized, interpersonal, and situational context.
  • Taxonomy Artifacts vs. Model Errors: Error analysis revealed that some "failures" are actually due to rigid annotation guidelines. For instance, AMCHA mandates that if an incident contains both expression and targeting, it must be labeled solely as Targeting. LLMs frequently flagged both, resulting in technical false positives despite being conceptually accurate.

Key Results

  • Binary Detection & False Positives: Both models performed well at binary classification. Notably, GPT-4o achieved a 0% false-positive rate on the Synthetic contrast dataset, whereas Llama-3.2-3B misclassified up to 13.36% of benign cultural events as antisemitic when prompted with definitions.
  • Fine-Grained Performance: GPT-4o significantly outperformed Llama-3.2-3B across all fine-grained tasks.
  • Prompting Impact: On the AMCHA dataset, adding definitions boosted GPT-4o's weighted F1 score for rhetoric types from 56% to 67%. For action types, adding in-context examples boosted the weighted F1 score from 56% to 66%.
  • Real-World Application: When deployed on the open-ended Campus-News dataset, GPT-4o acted as an effective initial filter, flagging 364 potential incidents out of 5,275 articles. However, it exhibited high recall but low precision, confirming its utility as a screening tool to assist—rather than replace—human analysts.

Limitations and Future Work

  • Selection and Reporting Bias: The datasets rely entirely on incidents that were formally reported or picked up by the media, which represents only a fraction of actual occurrences.
  • Geographic and Linguistic Constraints: The study is strictly limited to English-language, U.S.-focused contexts.
  • Lack of Fine-Tuning: The authors only explored prompting strategies. Future work should compare these results against task-specific fine-tuning, which might offer more robust performance at the cost of higher computational overhead.
  • Rigid Taxonomies: Current models struggle with mutually exclusive hierarchical categories when real-world events inherently blur those boundaries.

Summary updated

Abstract

Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.
BibTeX
@misc{halevy2026evaluating,
  author = {Karina Halevy and Julia Mendelsohn and Chan Young Park and Yulia Tsvetkov and Maarten Sap},
  title = {{Evaluating Large Language Models for Antisemitic Incident Classification}},
  year = {2026},
  eprint = {2607.04890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.04890},
}
Figure