Perspectivist Modeling of Human Label Variation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 250 papers60 conceptsfrozen 2026-10-04

In traditional machine learning, we treat human disagreement as noise. When three annotators disagree on whether a tweet is "toxic," the standard procedure is to take a majority vote or an average, creating a single "gold standard" label. This process assumes there is one objective truth and that any deviation from it is an error to be smoothed away.

However, for subjective tasks—like detecting sarcasm, identifying hate speech, or segmenting ambiguous medical images—disagreement is often a meaningful signal of task complexity or diverse cultural perspectives. A new research direction, often called Perspectivism, argues that we should stop aggregating labels and start modeling the individual "voices" within the crowd.

The Fallacy of the Gold Standard

The core motivation for this shift is the realization that "ground truth" is often a myth. Barbara Plank argues in The "Problem" of Human Label Variation that human label variation (HLV) is a valuable signal reflecting inherent ambiguity. When we collapse these perspectives into a single label, we lose the nuance required to handle real-world subjectivity.

In fact, forcing a consensus can actively harm model performance. Alexandros Potamianos and colleagues found in Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors that aggregated labels create "artifacts" that cause Large Language Models (LLMs) to ignore in-context evidence and fall back on their own pre-training biases. By contrast, training on raw, individual labels provides a cleaner, more consistent signal.

Architectures for Multiple Perspectives

To capture these individual signals, researchers have moved toward Multi-Task Annotator Modeling. Instead of a single output head, these models use a shared neural backbone (like BERT or a U-Net) connected to multiple independent classification heads—one for each human annotator.

  • Shared Representations: The model learns a universal understanding of the language or image features.
  • Individual Heads: Each head specializes in the specific biases, sensitivities, and "labeling style" of a single person.

Matthias Orlikowski and Philipp Heinisch identified an "architectural sweet spot" in Architectural Sweet Spots for Modeling Human Label Variation, showing that relating these perspectives to one another using recommender-system techniques can outperform majority-vote models by over 40%. This approach scales surprisingly well; Orlikowski and Dirk Hovy demonstrated that models can support thousands of unique annotator heads simultaneously (The Ecological Fallacy in Annotation).

Behavior Over Demographics

A common intuition is that we should model annotators using their demographics—age, gender, or political leanings. However, recent evidence suggests this is often a distraction. Matthias Orlikowski and Paul Röttger found that explicitly adding sociodemographic data does not significantly improve performance (The Ecological Fallacy in Annotation).

Instead, the most predictive signal is an individual's labeling behavior. Nikolas Vitsakis and Ioannis Konstas propose extracting "behavioral embeddings" from models trained to predict individual annotations (Voices in a Crowd). By clustering these embeddings, we can identify robust "voices" or schools of thought within a dataset without needing potentially biased metadata.

Why Perspectivism Works

Modeling individual variation provides three key advantages over traditional aggregation:

  1. Uncertainty Quantification: Multi-head models are significantly better at estimating uncertainty. If the individual heads disagree, the model knows the instance is inherently ambiguous. Xinpeng Wang used this in the ACTOR framework to strategically select which annotator to query, saving 70% of the annotation budget.
  2. Preserving Minority Voices: Majority voting silences minority perspectives. In tasks like hate speech detection, a "majority" might not find a statement offensive, but a specific marginalized group might. Perspectivist models, like those explored by Aiqi Jiang in Re-examining Sexism and Misogyny Classification, allow AI to account for these differing sensitivities.
  3. Robustness to Noise: By treating each worker as a separate task, models can distinguish between genuine disagreement and accidental error. Uthman Jinadu and Yi Ding 0010 developed a "subjectivity parameter" to clean up noisy datasets without erasing valid minority opinions (Noise Correction on Subjective Datasets).

The "perspectivist turn," as advocated by Federico Cabitza and Valerio Basile in Toward a Perspectivist Turn in Ground Truthing, represents a fundamental shift in AI. It moves us away from building systems that mimic a "generic" human average and toward systems that understand the messy, diverse, and inherently debatable nature of human interpretation.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 81 papers
  2. Version 1 · 2026-10-04 · 50 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_individual_annotator_modeling_adbd5dfa,
  title = {individual annotator modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/individual-annotator-modeling/rdr_adbd5dfa92cf45918881ae894d8346f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_perspectivist_modeling_of_human_label_va_adbd5dfa_v1,
  title = {Perspectivist Modeling of Human Label Variation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perspectivist-modeling-of-human-label-variation/rdr_adbd5dfa92cf45918881ae894d8346f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}