Learning from Human Label Variation in Subjective Tasks

This direction is split into other directions: subjective label ambiguity. Its last version stays readable and citable here.

Version 1 of 156 papers58 conceptsfrozen 2026-10-04

In traditional machine learning, we treat human disagreement as a bug. If five annotators look at a social media post and three label it "toxic" while two label it "safe," the standard pipeline collapses this nuance into a single "gold label" of toxicity. We treat the minority opinion as noise to be filtered out.

However, a growing body of research argues that for subjective tasks—like identifying hate speech, sarcasm, or social norms—this "myth of objectivity" is fundamentally flawed. As Zeerak Talat and colleagues argue in Disembodied Machine Learning: On the Illusion of Objectivity in NLP, correctness in social tasks is inherently tied to the observer's background. By forcing a consensus, we don't reach an objective truth; we simply silence minority perspectives and create models that are "overconfident liars," a concept explored by Agamdeep Singh in Soft-Label Training Preserves Epistemic Uncertainty.

The shift toward Data Perspectivism treats human disagreement not as noise, but as a vital signal.

The Problem with the Gold Standard

The core tension lies in how we conceptualize tasks. Paul Röttger, Dirk Hovy, and others distinguish between two paradigms in Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks:

  1. The Prescriptive Paradigm: Enforces a single policy to ensure model consistency. This is useful for legal or corporate moderation but fails to capture how people actually communicate.
  2. The Descriptive Paradigm: Captures a diverse survey of human beliefs. This acknowledges that "offensiveness" is an opinion, not a binary fact.

When we force subjective tasks into the prescriptive mold, we lose the "ecological validity" of the data. As Cecilia Ovesdotter Alm noted in her early call to action, Subjective Natural Language Problems, traditional loss functions are designed to converge on a single point. This creates a technical barrier for models that need to account for multiple valid interpretations of the same input.

Modeling the Distribution of Opinion

If we stop trying to find the "correct" label, what do we predict instead? One answer is Population-Level Learning (PLL). In Offensiveness as an Opinion: Dissecting population-level Label Distributions, Yu Liang and colleagues demonstrate that modeling the full distribution of human opinions provides a more nuanced reflection of toxicity than winner-take-all approaches.

To make this work technically, researchers like Uthman Jinadu and Yi Ding 0010 have developed frameworks that use a "subjectivity parameter." This allows a model to distinguish between genuine human disagreement (which should be preserved) and actual annotator error or noise (which should be corrected). Their work in Noise Correction on Subjective Datasets shows that cleaning data doesn't have to mean erasing minority voices.

From What to Who: Annotator-Centric Learning

A more radical approach involves modeling the annotators themselves. Instead of a single classification head, models like ACTOR, proposed by Xinpeng Wang and Barbara Plank, use multiple heads where each head represents a specific annotator. This allows the model to learn individual perspectives.

This shift changes how we collect data. Traditional Active Learning (AL) seeks consensus to find the "true" label. However, Anna Sotnikova and Hal Daumé III introduced Disagreement Aware Active Learning (DAAL) in Which Examples Should be Multiply Annotated?. DAAL targets the "Goldilocks zone": examples where the model’s uncertainty doesn't match the natural level of human disagreement.

Similarly, Annotator-Centric Active Learning (ACAL) moves beyond asking what data to label to asking who should label it, ensuring that the full spectrum of human judgment is captured efficiently.

Personalization and Context

The ultimate goal of this research direction is to move toward human-centered AI that understands individual context. Joan Plepi and Lucie Flek bridge this gap in Unifying Data Perspectivism and Personalization, showing that modeling an individual's history significantly improves predictions of how they will judge social conflicts.

By injecting personal profiles—personality traits, demographics, and values—into prompts, researchers like Yuxiang Zhou are helping Large Language Models (LLMs) move beyond predicting the "average" human reaction to simulating the subjective variance of real human experience (Modeling Subjectivity in Cognitive Appraisal with Language Models).

In summary, the "perspectivist turn" argued for by Federico Cabitza and Valerio Basile suggests that the future of AI lies not in finding a single objective truth, but in embracing the rich, messy, and valid diversity of human interpretation.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_from_human_label_variation_in_s_fc7b0b59,
  title = {Learning from Human Label Variation in Subjective Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-human-label-variation-in-subjective-tasks/rdr_fc7b0b59b1124c8fba20f06f2bbb5b86},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_from_human_label_variation_in_s_fc7b0b59_v1,
  title = {Learning from Human Label Variation in Subjective Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-human-label-variation-in-subjective-tasks/rdr_fc7b0b59b1124c8fba20f06f2bbb5b86/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}