In traditional machine learning, we treat human disagreement as a bug. If five annotators look at a social media post and three label it "toxic" while two label it "safe," the standard pipeline collapses this nuance into a single "gold label" of toxicity. We treat the minority opinion as noise to be filtered out.
However, a growing body of research argues that for subjective tasks—like identifying hate speech, sarcasm, or social norms—this "myth of objectivity" is fundamentally flawed. As Zeerak Talat and colleagues argue in Disembodied Machine Learning: On the Illusion of Objectivity in NLP, correctness in social tasks is inherently tied to the observer's background. By forcing a consensus, we don't reach an objective truth; we simply silence minority perspectives and create models that are "overconfident liars," a concept explored by Agamdeep Singh in Soft-Label Training Preserves Epistemic Uncertainty.
The shift toward Data Perspectivism treats human disagreement not as noise, but as a vital signal.
The Problem with the Gold Standard
The core tension lies in how we conceptualize tasks. Paul Röttger, Dirk Hovy, and others distinguish between two paradigms in Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks:
- The Prescriptive Paradigm: Enforces a single policy to ensure model consistency. This is useful for legal or corporate moderation but fails to capture how people actually communicate.
- The Descriptive Paradigm: Captures a diverse survey of human beliefs. This acknowledges that "offensiveness" is an opinion, not a binary fact.
When we force subjective tasks into the prescriptive mold, we lose the "ecological validity" of the data. As Cecilia Ovesdotter Alm noted in her early call to action, Subjective Natural Language Problems, traditional loss functions are designed to converge on a single point. This creates a technical barrier for models that need to account for multiple valid interpretations of the same input.
Modeling the Distribution of Opinion
If we stop trying to find the "correct" label, what do we predict instead? One answer is Population-Level Learning (PLL). In Offensiveness as an Opinion: Dissecting population-level Label Distributions, Yu Liang and colleagues demonstrate that modeling the full distribution of human opinions provides a more nuanced reflection of toxicity than winner-take-all approaches.
To make this work technically, researchers like Uthman Jinadu and Yi Ding 0010 have developed frameworks that use a "subjectivity parameter." This allows a model to distinguish between genuine human disagreement (which should be preserved) and actual annotator error or noise (which should be corrected). Their work in Noise Correction on Subjective Datasets shows that cleaning data doesn't have to mean erasing minority voices.
From What to Who: Annotator-Centric Learning
A more radical approach involves modeling the annotators themselves. Instead of a single classification head, models like ACTOR, proposed by Xinpeng Wang and Barbara Plank, use multiple heads where each head represents a specific annotator. This allows the model to learn individual perspectives.
This shift changes how we collect data. Traditional Active Learning (AL) seeks consensus to find the "true" label. However, Anna Sotnikova and Hal Daumé III introduced Disagreement Aware Active Learning (DAAL) in Which Examples Should be Multiply Annotated?. DAAL targets the "Goldilocks zone": examples where the model’s uncertainty doesn't match the natural level of human disagreement.
Similarly, Annotator-Centric Active Learning (ACAL) moves beyond asking what data to label to asking who should label it, ensuring that the full spectrum of human judgment is captured efficiently.
Personalization and Context
The ultimate goal of this research direction is to move toward human-centered AI that understands individual context. Joan Plepi and Lucie Flek bridge this gap in Unifying Data Perspectivism and Personalization, showing that modeling an individual's history significantly improves predictions of how they will judge social conflicts.
By injecting personal profiles—personality traits, demographics, and values—into prompts, researchers like Yuxiang Zhou are helping Large Language Models (LLMs) move beyond predicting the "average" human reaction to simulating the subjective variance of real human experience (Modeling Subjectivity in Cognitive Appraisal with Language Models).
In summary, the "perspectivist turn" argued for by Federico Cabitza and Valerio Basile suggests that the future of AI lies not in finding a single objective truth, but in embracing the rich, messy, and valid diversity of human interpretation.