Embracing Annotator Disagreement in Toxicity Detection

288 concepts208 papers

In traditional supervised learning, we treat human disagreement as noise. If three annotators look at an image of a cat, and two say "cat" while one says "dog," we assume the third annotator made a mistake. We take the majority vote, assign a hard "gold standard" label, and train our models to minimize cross-entropy loss against it.

For years, natural language processing applied this exact same pipeline to content moderation tasks like hate speech, sexism, and aggression detection. But language is not an image of a cat. What one person considers a harmless joke, another perceives as a toxic insult.

When we force a majority consensus on highly subjective text, we do not eliminate noise—we erase crucial social and cultural context. A growing body of research demonstrates that inter-annotator disagreement is not a bug to be cleaned out of datasets; it is a fundamental feature of human language that models must learn to navigate.

The Breakdown of Consensus

Why do human evaluators disagree so frequently on toxic content? The breakdown in agreement typically happens along three linguistic fault lines: reappropriation, coded rhetoric, and shifting legal or cultural thresholds.

Context and Reappropriation

Linguistic terms frequently change meaning depending on the identity of the speaker and the audience. For instance, Irene Kwok et al. demonstrated in Locate the Hate: Detecting Tweets against Blacks that while certain slurs are unambiguous indicators of racism when used externally, their reappropriation within specific communities makes automated, keyword-based detection highly unreliable. Similarly, Juan Vásquez et al. found in HOMO-MEX: A Mexican Spanish Annotated Corpus for LGBT+phobia Detection on Twitter that distinguishing between genuine phobic attacks and internal community irony or humor creates severe ambiguity, leading to low inter-annotator agreement even among individuals familiar with the targeted group.

Coded Rhetoric and Intent

Harmful speech rarely announces itself explicitly. Speakers frequently employ dual-meaning expressions to mask their intent. Ronan Le Bras explores this in From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models, showing how "dogwhistles" allow individuals to signal provocative or hateful messages to an in-group while maintaining a literal, benign interpretation that shields them from moderation. When annotators lack specific in-group knowledge, they rate these statements as entirely neutral, fracturing the ground truth. Furthermore, as Sanjana Sharma et al. note in Degree based Classification of Harmful Speech using Twitter Data, assessing the degree of harmful intent—separating casual trolling from an explicit incitement to violence—is inherently subjective, yielding moderate agreement at best.

Even experts cannot agree on where the boundary lies. When Vincent Kums et al. tasked legal experts and students with labeling content based on actual statutory definitions in A Novel Dataset for Classifying German Hate Speech Comments with Criminal Relevance, they found that distinguishing between legally protected offensive speech and criminally punishable hate speech remains deeply context-dependent and subjective, defying objective standardization.

The Cost of Forcing Consensus

When datasets flatten this variance through majority voting, it severely damages a model's ability to generalize.

Consider what happens during training. If a sentence receives two "toxic" labels and one "safe" label, a majority-vote framework treats it as 100% toxic. The model is penalized if it outputs a probability of 0.66, even though that probability perfectly reflects the human distribution.

The authors of MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection found that forcing models to learn a single, flattened ground truth on subjective tasks like sexism detection causes "lexical overfitting." Instead of learning the nuanced, contextual dynamics of language, the model simply memorizes specific high-frequency keywords or slurs. When deployed in the real world, these models fail: they flag benign discussions containing reclaimed language and miss subtle, implicit attacks entirely.

As Nauros Romim et al. argue in BD-SHS: A Benchmark Dataset for Learning to Detect Online Bangla Hate Speech in Different Social Contexts, we must transition away from consensus-enforcing models toward interpretive models. Instead of asking "Is this text toxic?" and demanding a binary answer, we must design systems that understand why different groups perceive the text differently.

Algorithmic Paths Forward

To build models that handle the inherent subjectivity of language, researchers are changing both how data is structured and how models reason.

Multi-Agent Negotiation and Soft Labels

Rather than relying on a static, single-point label, recent frameworks simulate the disagreement process itself using Large Language Models (LLMs).

In PREDICT: Multi-Agent-based Debate Simulation for Generalized Hate Speech Detection, Someen Park et al. introduced a framework where multiple LLM agents are assigned distinct perspectives based on different benchmark datasets. When presented with an ambiguous text, the agents debate its meaning. This simulation allows the system to capture conflicting definitions and achieve significantly better generalization than traditional majority-voting models.

[Ambiguous Input Text]
          │
          ▼
┌─────────────────────────────────────────────────────────┐
│               Multi-Agent Debate (PREDICT)              │
│                                                         │
│  ┌──────────────────┐             ┌──────────────────┐  │
│  │  Agent A (Strict)│ ◄─────────► │ Agent B (Lenient)│  │
│  └─────────┬────────┘             └─────────┬────────┘  │
└────────────┼────────────────────────────────┼───────────┘
             │                                │
             ▼                                ▼
     [Soft Label: 0.70]               [Soft Label: 0.30]
             │                                │
             └────────────────┬───────────────┘
                              ▼
                [Final Distributional Target]

Similarly, Han Wang et al. developed a self-training framework detailed in Multi-Agent VLMs Guided Self-Training with PNU Loss for Low-Resource Offensive Content Detection. By pairing a strict moderator agent with a lenient user agent, the system negotiates labels for ambiguous content. Combined with a specialized Positive-Negative-Unlabeled (PNU) loss function, this approach allows smaller, downstream models to learn from the distribution of the disagreement rather than a brittle binary choice.

Hierarchical and Token-Level Schemas

Another way to preserve context is to change the granularity of the annotation schema. If annotators are forced to give a flat binary label to a complex sentence, they will disagree. However, if the schema breaks the task down into sub-components, consensus improves, and the model gains better signal.

  • Hierarchical Labeling: Paula Fortuna et al. proposed a fine-grained, hierarchical scheme in A Hierarchically-Labeled Portuguese Hate Speech Dataset. By separating the target, the type of discrimination, and the explicit strategy used, the authors captured the intersections of hate speech categories, making the underlying subjectivity measurable and actionable for the model.
  • Token-Level Slot Filling: Moving beyond sentence-level labels entirely, Usman Naseem et al. introduced token-level annotation in GameTox: A Comprehensive Dataset and Analysis for Enhanced Toxicity Detection in Online Gaming Communities. By labeling specific game-specific slang and verbs alongside the overall intent, the model learns exactly which linguistic anchors trigger disagreement, leading to a more context-aware understanding of online harassment.

By treating annotator variance as a rich distribution of human perspectives rather than an error metric, these approaches allow machine learning systems to move past rigid, fragile classification and toward a more realistic, nuanced understanding of human discourse.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_annotation_subjectivity_inter_annotator__5af52033,
  title = {annotation subjectivity (inter-annotator agreement challenges)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-subjectivity-inter-annotator-agreement-challenges/rdr_5af52033b2514f2c9c7953e8e417774a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_annotation_subjectivity_inter_annotator__5af52033_v1,
  title = {annotation subjectivity (inter-annotator agreement challenges)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-subjectivity-inter-annotator-agreement-challenges/rdr_5af52033b2514f2c9c7953e8e417774a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake