Modeling Human Label Variation as a Structured Signal

35 concepts31 papers1 proposal

In traditional machine learning, the path to a high-quality dataset is paved with consensus. Researchers typically collect multiple human annotations for a single data point and then aggregate them—often through majority voting or averaging—to produce a single "gold standard" label. This process rests on a fundamental assumption: that there is one objective truth, and any deviation from it is merely statistical noise or human error.

However, for tasks involving human values, aesthetics, or social nuances, this assumption breaks down. As Cecilia Ovesdotter Alm argues in Subjective Natural Language Problems: Motivations, Applications, Characterizations, and Implications, treating annotator disagreement as noise to be minimized ignores the meaningful signal regarding the inherent complexity of the language phenomenon. When two people disagree on whether a comment is "offensive" or a song belongs to a specific "genre," they are often not making a mistake; they are expressing valid, differing interpretations rooted in their unique cultural and individual perspectives.

The Perspectivist Turn

A growing body of research advocates for a "perspectivist turn," moving away from the search for a single ground truth toward a framework that embraces data diversity. Julia Romberg et al. suggest in Towards a Perspectivist Turn in Argument Quality Assessment that argument quality is not an inherent property of text but depends entirely on who is doing the judging. By preserving individual disagreements rather than averaging them away, models can learn to understand multiple perspectives simultaneously.

This shift is particularly critical in sensitive domains like morality and hate speech. Ines Reinig et al. note in A Survey on Modelling Morality for Text Analysis that low inter-annotator agreement is a meaningful signal of moral pluralism. Similarly, Paul Röttger et al. distinguish between prescriptive annotation (enforcing a single policy) and descriptive annotation (capturing a diversity of human beliefs), arguing that the latter is essential for building models that are truly representative of a population.

From Majority Vote to Soft Labels

If we stop collapsing labels into a single category, how do we train models? One effective mechanism is the use of soft labels, where the model is trained to predict the full distribution of human opinions rather than just the majority vote.

The SemEval-2023 Task 11: Learning with Disagreements (LeWiDi) competition demonstrated that models predicting these distributions are more robust and better at capturing nuance. Tommaso Fornaciari et al. further showed in Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning that a multi-task learning framework—predicting both the "correct" label and the distribution of opinions—reduces overfitting on ambiguous data. By treating disagreement as a "treasure" of information rather than a nuisance, as Caixia Zhou et al. describe in the context of edge detection, models can prioritize hard-to-classify regions that contain the most structural complexity.

Modeling the Annotator

Beyond just predicting distributions, researchers are now modeling the annotators themselves. Instead of treating workers as interchangeable units, frameworks like CrowdWorkSheets encourage documenting the individual and collective identities underlying annotations.

Advanced architectures are being developed to relate these identities to the labels they produce:

  • Individual-Aware Modeling: Philipp Heinisch et al. found that using recommender-system techniques to relate different perspectives allows models to predict when and why a specific individual will deviate from the majority.
  • Demographic Alignment: Eve Fleisig et al. developed models that predict ratings based on annotator demographics, allowing systems to prioritize the perspectives of groups most affected by specific content, such as targets of hate speech.
  • Bias Modeling: Trevor Cohn and Lucia Specia demonstrated that explicitly modeling individual annotator biases using Multi-task Gaussian Processes significantly improves accuracy and data efficiency.

Why It Works: Intuition

The intuition behind this research is that disagreement is structured. Human variation is rarely random; it follows patterns dictated by culture, experience, and the inherent ambiguity of the stimulus.

For example, in End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks, Navin Raj Prabhu et al. show that by training on the full distribution of emotional ratings, a model can learn to predict not just the intensity of an emotion, but also the level of human uncertainty associated with it. This allows the model to signal when a situation is truly ambiguous.

By preserving this variance, we avoid the "consensus problem" identified by Alastair J. D. Craft et al., where valid cultural responses are discarded. Ultimately, as Kelsey Kraus and Margaret Kroll argue, preserving the full distribution of opinions allows AI to better align with the nuanced, diverse expectations of a varied human user base.

Key Foundational Work

Subjective Natural Language Problems: Motivations, Applications, Characterizations, and Implications (2011) — contributed 2 concept elements to this direction.
How Many Beans Make Five? The Consensus Problem in Music-Genre Classification and a New Evaluation Method for Single-Genre Categorisation Systems (2007) — contributed 1 concept elements to this direction.
Semantic multimedia (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subjective_label_variance_98aef799,
  title = {subjective label variance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-label-variance/rdr_98aef799aa744ce19476a126de878d1c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_subjective_label_variance_98aef799_v1,
  title = {subjective label variance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-label-variance/rdr_98aef799aa744ce19476a126de878d1c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake