Who Speaks Matters: Analysing the Influence of the Speaker's Ethnicity on Hate Classification

Ananya Malik, Kartik Sharma, Shaily Bhatt, Lynnette Hui Xian Ng

TL;DR

Deploying Large Language Models (LLMs) for automated content moderation seems efficient, but how robust are they to cultural and regional variations in speech? This paper demonstrates that injecting a speaker's linguistic identity—either explicitly by stating their nationality or implicitly through regional dialects—causes models to frequently flip their hate speech classifications. Alarmingly, implicit dialect markers significantly increase false negatives, allowing actual hate speech to slip through undetected.

What Problem Does This Paper Address?

Automated hate speech detection is a high-stakes task where errors can either silence marginalized communities (false positives) or allow toxic behavior to proliferate (false negatives). While LLMs perform well on standard benchmarks, they are heavily biased toward Standard American English.

The authors investigate a critical gap: How robust are these models when the inputs contain markers of the speaker's linguistic identity? If a user writes a comment using African-American English, British English, Indian English, Jamaican Patois, or Singaporean English (Singlish), does the model's classification change even if the core semantic meaning remains the same?

Methods

The researchers evaluated the robustness of four models: a fine-tuned HateXplain-BERT model, Llama-3-8B, Llama-3-70B, and GPT-4o. They tested two prompting strategies: zero-shot classification and in-context learning (ICL) with four examples.

To evaluate the impact of linguistic identity, they modified baseline sentences using two types of markers:

  1. Explicit Markers: They prepended a phrase indicating the speaker's background (e.g., The [ethnicity] person said, "[input]").
  2. Implicit Markers: They transformed the baseline text into specific regional dialects (e.g., adding colloquialisms, code-mixing, or region-specific spellings) using a few-shot Llama-3-70B model, followed by human verification to ensure semantic meaning was preserved.
Figure 1: We investigate whether adding the identity of the speaker, whether Singaporean or Jamaican, can affect the model’s hate speech classification on the same sentence. Our findings indicate that model outputs do flip because of the presence of such markers, and the percentage of flips depends on the marker, model size, and the ethnicity injected.

The core metric tracked was the flip percentage—how often a model changed its prediction (from hateful to non-hateful, or vice versa) after an identity marker was introduced.

Datasets

The study utilized two public hate speech benchmarks:

  • HateXplain: A dataset derived from Twitter and Gab. The authors sampled 3,000 sentences (2,094 Hateful, 906 Non-Hateful).
  • MPBHSD (Multi-Platform Bias-Mitigated Hate Speech Dataset): Sourced from Twitter, 4chan, and Reddit, comprising 600 Hateful and 2,400 Non-Hateful sentences.

Key Insights and Ideas

  • Implicit Dialects Break Models More Than Explicit Labels: Models are relatively stable when told "A Jamaican person said X." However, when the text is actually written in Jamaican dialect, the classification accuracy degrades sharply.
  • The Asymmetry Danger (More False Negatives): When predictions flipped, they overwhelmingly flipped from Hateful to Non-Hateful (H $\rightarrow$ NH). This means that when hate speech is spoken in a non-standard dialect, LLMs fail to recognize it, creating a severe safety loophole.
  • In-Context Learning (ICL) Acts as a Stabilizer: Providing few-shot examples in the prompt consistently lowered the flip percentage compared to zero-shot prompting, suggesting that examples help anchor the model's decision boundaries across different dialects.
  • Model Size Correlates with Robustness: Larger, newer models like Llama-3-70B and GPT-4o exhibited significantly fewer flips than the smaller Llama-3-8B model, though they were still far from immune.

Key Results

Without any identity markers, the models achieved high baseline performance (up to 90% accuracy on MPBHSD and 80% on HateXplain). However, the introduction of implicit dialect markers caused significant performance degradation:

  • HateXplain-BERT saw its predictions flip up to 43.4% of the time on the HateXplain dataset when implicit dialect features were introduced.
  • Llama-3-8B (ICL) proved highly unstable on the MPBHSD dataset, with flip rates hovering around 40% across all tested dialects.
  • Even GPT-4o (Zero-shot) experienced a 26.96% flip rate on African-American dialect inputs within the HateXplain dataset.
Bar charts illustrating the percentage of prediction flips across different models and dialects.

Statistical validations (t-tests and McNemar's tests) confirmed that the speaker's linguistic identity significantly alters classification outcomes ($p < 0.05$).

Limitations and Future Work

The authors note several limitations in their current work:

  • Synthetic Dialect Data: Due to a lack of real-world, human-annotated hate speech datasets in multiple dialects, the implicit text had to be synthetically generated via an LLM. While human-verified, this approach may introduce synthetic artifacts.
  • English-Centric Focus: The study is limited to English-based dialects and does not explore the complexities of fully multilingual hate speech detection.
  • Model Constraints: Highly guarded commercial models like Claude were not evaluated due to resource constraints and safety filter refusals during testing.

Future research should focus on developing more interpretable evaluation frameworks to pinpoint exactly which dialectal phrases trigger these classification failures.

Summary updated

Abstract

Large Language Models (LLMs) offer a lucrative promise for scalable content moderation, including hate speech detection. However, they are also known to be brittle and biased against marginalised communities and dialects. This requires their applications to high-stakes tasks like hate speech detection to be critically scrutinized. In this work, we investigate the robustness of hate speech classification using LLMs particularly when explicit and implicit markers of the speaker's ethnicity are injected into the input. For explicit markers, we inject a phrase that mentions the speaker's linguistic identity. For the implicit markers, we inject dialectal features. By analysing how frequently model outputs flip in the presence of these markers, we reveal varying degrees of brittleness across 3 LLMs and 1 LM and 5 linguistic identities. We find that the presence of implicit dialect markers in inputs causes model outputs to flip more than the presence of explicit markers. Further, the percentage of flips varies across ethnicities. Finally, we find that larger models are more robust. Our findings indicate the need for exercising caution in deploying LLMs for high-stakes tasks like hate speech detection.
BibTeX
@inproceedings{malik2024who,
  author = {Ananya Malik and Kartik Sharma and Shaily Bhatt and Lynnette Hui Xian Ng},
  title = {{Who Speaks Matters: Analysing the Influence of the Speaker's Ethnicity on Hate Classification}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=UCLQTzpJ7k},
}
Figure