Modeling Human Label Variation in Subjective Tasks

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2252 papers348 conceptsfrozen 2026-10-04

In traditional supervised learning, we treat data annotation as a search for a single "ground truth." When multiple human annotators disagree on a label, the standard procedure is to aggregate their opinions through majority voting or adjudication. This process assumes that disagreement is merely "noise"—a statistical error to be filtered out to reach the "true" signal.

However, for subjective tasks like hate speech detection, humor recognition, or aesthetic assessment, this assumption fails. As Paul Röttger and colleagues argue in Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks, disagreement is often a meaningful signal of cultural, ideological, or personal diversity. By forcing a consensus, we don't just clean the data; we systematically erase minority perspectives and bake majority bias into our models.

The Fallacy of the Gold Standard

The "Gold Standard" approach treats human perception as a deterministic measurement. But in social contexts, "truth" is often a distribution rather than a point. Kofi Arhin demonstrates in Ground-Truth, Whose Truth? that toxicity is inherently context-dependent. When we use majority voting to label toxic text, we often silence the very marginalized groups who are the primary targets of that speech, as their lived experiences lead them to perceive harm that a dominant majority might overlook.

This "mean-seeking bias" is exacerbated by automated quality filters. Dan Klein 0001 and Philipp Cimiano found that aggressive filtering based on inter-annotator agreement metrics often mistakes valid minority opinions for spam, resulting in datasets that lack demographic diversity (Balancing Quality and Variation).

From Consensus to Perspectivism

To address this, a new research direction—often called Data Perspectivism—proposes modeling the full spectrum of human disagreement. Instead of predicting a single label, models are designed to capture the variation in how different people perceive the same input.

Several mechanisms have emerged to implement this:

  1. Annotator Embeddings: Rather than treating annotators as anonymous units, researchers like Naihao Deng and Lu Wang use learnable embeddings to represent individual annotator idiosyncrasies (You Are What You Annotate). This allows a model to predict how a specific person (or demographic group) might label a piece of content.
  2. Soft Labels and Distribution Estimation: Instead of a hard 0 or 1, models are trained to predict the "soft" distribution of crowd opinions. The SemEval-2023 LeWiDi task, led by Elisa Leonardelli and Massimo Poesio, formalized this by challenging teams to predict the full probability distribution of human judgments.
  3. Evidential Modeling: To distinguish between "uncertainty because the task is hard" and "uncertainty because people disagree," Wen Wu and Chao Zhang utilize Deep Evidential Regression. This allows models to quantify their own epistemic and aleatoric uncertainty, reflecting the inherent subjectivity of tasks like emotion recognition (Estimating the Uncertainty in Emotion Attributes).

Architectural Sweet Spots

How do we relate these different perspectives without becoming overwhelmed by the number of individual annotators? Philipp Heinisch and Matthias Orlikowski identified an "architectural sweet spot" by using recommender-system techniques. By relating different perspectives to one another, their models can generalize across annotators, outperforming traditional majority-vote models by significant margins (Architectural Sweet Spots for Modeling Human Label Variation).

Similarly, the ARTICLE framework by Sujan Dutta and Ashiqur R. KhudaBukhsh uses LLMs to measure an annotator's internal consistency rather than their agreement with the majority. This preserves unique but consistent minority voices while still filtering out truly erratic, low-quality labels (Annotator Reliability Through In-Context Learning).

The Social and Ethical Imperative

Moving beyond majority voting is not just a technical optimization; it is a matter of "design justice." Paula Fortuna and Zeerak Talat argue that treating hate speech as a standard classification task ignores its deeply political nature (Directions for NLP Practices Applied to Online Hate Speech Detection).

When we collapse diverse annotations into a single "aligned" behavior, we risk what Hannah Rose Kirk calls "digital colonialism"—imposing the narrow values of a specific developer group onto a global user base (The Empty Signifier Problem). By embracing disagreement as a feature, researchers are building AI that is more representative, more robust, and more capable of navigating the complex, subjective reality of human communication.

Papers in this version

Showing 60 of 252 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 207 papers
  2. Version 1 · 2026-10-04 · 252 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_minority_perspective_erasure_fbe1af68,
  title = {minority perspective erasure},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minority-perspective-erasure/rdr_fbe1af68cc3d49a0aa7987efb85edf0e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_human_label_variation_in_subjec_fbe1af68_v1,
  title = {Modeling Human Label Variation in Subjective Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-human-label-variation-in-subjective-tasks/rdr_fbe1af68cc3d49a0aa7987efb85edf0e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}