Distinguishing Use from Mention in Toxicity Detection

48 concepts34 papers1 proposal

Automated toxicity and hate speech detection models often boast high accuracy on standard evaluation benchmarks. However, when deployed in the wild, their performance frequently collapses. As Manuel Tonneau et al. demonstrated in HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter, academic benchmarks can overestimate real-world performance by a factor of four.

Why do these models fail so dramatically? The root cause lies in a reliance on lexical shortcuts. Instead of understanding the underlying intent, pragmatics, or target of a statement, models over-sensitize to specific toxic keywords, slurs, and profanity. This creates a severe systemic flaw: the inability to separate the active perpetration of hate speech from benign or even constructive expressions that happen to contain the same vocabulary.

The Use-Mention Error and Meta-Discourse

The most prominent failure mode occurs when a user references a toxic term without endorsing it. In linguistics, this is known as the use-mention distinction. A model that lacks this distinction cannot tell the difference between a perpetrator using a slur to harm someone and an activist mentioning that same slur to condemn, document, or debunk it.

As Kristina Gligoric et al. showed in NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps, even state-of-the-art large language models frequently fail this test. This technical limitation leads directly to the accidental censorship of counter-speech. When an individual stands up to abuse, they often quote or reference the original toxic language to refute it. Because standard classification models rely on isolated text analysis or keyword matching, they conflate the mention of harmful terms with the intent to harm, incorrectly flagging the defender as the aggressor (Bertie Vidgen et al. in Introducing CAD: the Contextual Abuse Dataset).

                  [ Input Text ]
                        │
         "They called her a [SLUR]!"
                        │
         ┌──────────────┴──────────────┐
         ▼                             ▼
  [ Lexical Shortcut ]        [ Pragmatic Context ]
  • Sees "[SLUR]"             • Identifies reporting intent
  • Triggers high toxicity    • Recognizes meta-discourse
         │                             │
         ▼                             ▼
❌ MISCLASSIFICATION               CORRECT
   (Flags as Hate Speech)        (Flags as Safe)

This issue extends to victims reporting their experiences or researchers documenting online harassment. Alberto Barrón-Cedeño et al. noted that models struggle to separate active abuse from narrative accounts of abuse, trapping victims in a "subjectivity trap" where their descriptions of harassment are misclassified as the harassment itself (UniBO @ AMI: A Multi-Class Approach to Misogyny and Aggressiveness Identification on Twitter Posts Using AlBERTo). Similarly, Michael Wiegand et al. argued in Implicitly Abusive Language - What does it actually look like and why are we not getting there? that toxicity detection models consistently fail to distinguish primary expressions of hate speech from meta-discourse due to these surface-level lexical shortcuts.

Conflating Profanity with Targeted Hate

A parallel failure mode is the inability to distinguish between general offensive language (such as profanity, venting, or aggressive banter) and targeted hate speech directed at protected groups.

In natural social media environments, general profanity is highly prevalent. When models are trained primarily on datasets where toxic words heavily correlate with hate speech, they become over-sensitized to vulgarity (Manuel Tonneau et al., HateDay). Consequently, when a user expresses intense frustration about an abstract concept, policy, or event using strong language, the model flags it as hate speech, failing to recognize that the hostility lacks a protected target (Marek Suppa et al. in Bryndza at ClimateActivism 2024: Stance, Target and Hate Event Detection via Retrieval-Augmented GPT-4 and LLaMA).

To build effective moderation systems, models must separate personal expression or general aggression from identity-based harassment. As Shivang Chopra et al. observed in Hindi-English Hate Speech Detection: Author Profiling, Debiasing, and Practical Perspectives, relying solely on profanity lexicons produces high false-positive rates because offensive language is frequently used for non-malicious personal expression. This boundary becomes even fuzzier in competitive subcultures—such as sports discussions—where aggressive "trash talk" is normalized but must be distinguished from dehumanizing abuse (Edinam Kofi Klutse et al. in Dismantling Hate: Understanding Hate Speech Trends Against NBA Athletes).

Why Models Fall into the Shortcut Trap

Why do models default to these brittle keyword strategies? The issue stems from how models learn from training data. If a training dataset contains a high density of specific slurs or profane words exclusively within its "hate speech" class, the model learns that the presence of the word is the classification criterion, completely bypassing the pragmatic context.

To expose these hidden flaws, Paul Rottger et al. introduced HateCheck: Functional Tests for Hate Speech Detection Models. By moving away from aggregate metrics like $F_1$ scores and instead testing models on targeted linguistic variations (such as negation, reclaimed slurs, and counter-speech), the authors systematically demonstrated that state-of-the-art models fail on basic contextual nuances. When a hateful statement is negated or framed within a counter-speech response, the model's accuracy plummets because it cannot look past the toxic linguistic features to see the overarching intent.

Strategies for Disentangling Context

Resolving this over-sensitization requires moving beyond simple text classification architectures toward models that explicitly capture context and intent. Several approaches have emerged across the literature:

  • Contextual and Hierarchical Taxonomy: Datasets like the Contextual Abuse Dataset (CAD) incorporate conversational context and human-highlighted rationales, forcing models to evaluate text within its dialogue thread rather than in isolation (Bertie Vidgen et al., Introducing CAD).
  • Adversarial Debiasing: Techniques designed to explicitly decouple the presence of offensive language from the hate speech class help models learn more robust decision boundaries (Shuzhou Yuan et al. in Separating Hate Speech and Offensive Language Classes via Adversarial Debiasing).
  • Explicit Prompting and Disambiguation: For large language models, leveraging explicit reasoning steps—such as Chain-of-Thought prompting to force the model to evaluate whether a term is being used or mentioned—has been shown to drastically reduce false-positive censorship errors (Kristina Gligoric et al., NLP Systems That Can't Tell Use from Mention).

Ultimately, building safe and equitable content moderation systems requires moving past the illusion of high benchmark scores. Models must transition from simple keyword detectors to systems capable of navigating the complex pragmatic landscape of human discourse, ensuring that those who counter or report abuse are not silenced by the tools meant to protect them.

Key Foundational Work

Interpreting Neural Network Hate Speech Classifiers (2018) — contributed 1 concept elements to this direction.
Relating Romanized Comments to News Articles by Inferring Multi-Glyphic Topical Correspondence (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_counter_speech_misclassification_8ba0ee19,
  title = {counter-speech misclassification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/counter-speech-misclassification/rdr_8ba0ee1958e34c949e04ea21c79aca5a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_counter_speech_misclassification_8ba0ee19_v1,
  title = {counter-speech misclassification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/counter-speech-misclassification/rdr_8ba0ee1958e34c949e04ea21c79aca5a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake