Evaluating Large Language Models for Detecting Antisemitism

Jay Patel, Hrudayangam Mehta, Jeremy Blackburn

TL;DR

This paper introduces "Guided-CoT," a novel prompting technique that significantly improves the ability of large language models (LLMs) to detect antisemitism by mimicking human annotators' thought processes. The study demonstrates that Guided-CoT outperforms standard prompting methods across various LLMs, offering a more reliable and interpretable approach to content moderation for sensitive topics.

What Problem Does This Paper Address?

Detecting hate speech, particularly antisemitism, on social media is a critical but challenging task. While LLMs show promise, their effectiveness is limited by the need for continuous adaptation to evolving language and the complexity of moderation policies. Existing methods like fine-tuning require extensive labeled data and struggle with scalability. This paper addresses the need for robust, adaptable, and interpretable LLM-based solutions for antisemitism detection, focusing on leveraging in-context learning with explicit policy guidelines.

Methods

The core contribution is the Guided-CoT prompting technique. Unlike standard Chain-of-Thought (CoT) prompts that simply ask models to "think step by step," Guided-CoT breaks down the reasoning process into specific, human-like steps. These steps include:

  1. Chunking and explaining content: Deconstructing the post to understand its components.
  2. Identifying linguistic cues: Detecting sarcasm, criticism, coded language, and other subtle tones.
  3. Determining intent: Differentiating between promoting antisemitism and raising awareness.
  4. Aligning with policy: Evaluating the post against the IHRA definition and its examples.
  5. Assessing author stance: Judging the author's favorability towards the Jewish community.

The paper also explores other prompting techniques like Zero-Shot (ZS), Zero-Shot-CoT (ZS-CoT), and variants of ZS that incorporate the IHRA definition (ZS-a, ZS-β). They also employ decoding strategies like greedy decoding and self-consistency to evaluate model robustness.

Datasets

The study utilizes a dataset of 11,315 Twitter posts collected between January 2019 and April 2023, labeled by human annotators based on the IHRA definition. This dataset consists of 9,362 non-antisemitic and 1,953 (17%) antisemitic posts. The authors note that this dataset is a subset of a larger dataset used in prior work (Jikeli et al., 2023a, 2024b).

Key Insights and Ideas

  • Guided-CoT is a game-changer: The proposed Guided-CoT prompting significantly boosts performance across all evaluated LLMs, demonstrating the power of structured, human-like reasoning for complex classification tasks.
  • Context matters, but not always as expected: Simply adding more context (like the IHRA definition with examples) to Zero-Shot prompts doesn't always improve performance; in some cases, it can even decrease it. Guided-CoT, however, effectively leverages this context.
  • LLMs struggle with nuance: Despite advancements, LLMs still misclassify posts due to misinterpreting sarcasm, coded language, quoted text, or even typos that resemble slurs. This highlights the ongoing challenge of contextual understanding.
  • Model explanations reveal paradoxes: The study introduces metrics to quantify semantic divergence in LLM explanations, revealing that some models exhibit paradoxical behaviors where their explanations for antisemitic and non-antisemitic content are surprisingly similar, or vice-versa.
  • Llama 3.1 70B is a strong performer: Notably, Llama 3.1 70B (even a quantized version) outperforms a fine-tuned GPT-3.5 on this task, showcasing the capabilities of open-source models.

Key Results

  • Performance Boost: Guided-CoT consistently improved F1 scores across models, with improvements ranging from 0.03 to 0.13 compared to Zero-Shot-CoT when using self-consistency. Refusal rates were reduced to nearly 0% for some models.
  • Llama 3.1 70B vs. GPT-3.5: Llama 3.1 70B with Guided-CoT achieved an F1 score of 0.72 (greedy) and 0.73 (self-consistency), outperforming fine-tuned GPT-3.5 (F1=0.70) on a comparable dataset.
  • Model Variability: Different models exhibit distinct patterns in invalid responses, refusal rates, and explanation styles. Gemma models often produced indeterminate classifications, while Llama 8B had a higher refusal rate.
  • Misclassification Analysis: 28% of misclassified posts fell under "Tone/Language/Stereotypes," and 25% under "Quotes/Reporting." Common errors included misinterpreting criticism of Israel as antisemitism and flagging typos like "kikes" for "likes."

Limitations and Future Work

  • Dataset Scope: The study is limited to English-language Twitter posts, potentially limiting generalizability to multilingual settings or other platforms (e.g., Reddit, YouTube).
  • Contextual Ambiguity: Annotators were allowed to use additional context during labeling, which might not be fully replicated by LLMs, leading to discrepancies.
  • Hyperparameter Tuning: The study used a single set of hyperparameters for sampling and self-consistency, acknowledging that alternative configurations could yield slightly different results.
  • Model Generalizability: Guided-CoT was evaluated on a specific task (antisemitism detection) and dataset; its effectiveness on other downstream tasks or different moderation policies needs further exploration.
  • Future Work: The authors suggest using these findings to audit models qualitatively, enhancing understanding of limitations and biases. Further research could explore multimodal contexts, multilingual settings, and a wider range of moderation policies.

Summary updated

Abstract

Detecting hateful content is a challenging and important problem. Automated tools, like machine-learning models, can help, but they require continuous training to adapt to the ever-changing landscape of social media. In this work, we evaluate eight open-source LLMs' capability to detect antisemitic content, specifically leveraging in-context definition. We also study how LLMs understand and explain their decisions given a moderation policy as a guideline. First, we explore various prompting techniques and design a new CoT-like prompt, Guided-CoT, and find that injecting domain-specific thoughts increases performance and utility. Guided-CoT handles the in-context policy well, improving performance and utility by reducing refusals across all evaluated models, regardless of decoding configuration, model size, or reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5. Additionally, we examine LLM errors and introduce metrics to quantify semantic divergence in model-generated rationales, revealing notable differences and paradoxical behaviors among LLMs. Our experiments highlight the differences observed across LLMs' utility, explainability, and reliability. Code and resources available at: https://github.com/idramalab/quantify-llm-explanations
BibTeX
@inproceedings{patel2025evaluating,
  author = {Jay Patel and Hrudayangam Mehta and Jeremy Blackburn},
  title = {{Evaluating Large Language Models for Detecting Antisemitism}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure