Data Perspectivism for Subjective Natural Language Processing

26 concepts24 papers1 proposal

Traditional machine learning treats human disagreement as noise. If three annotators label a sentence as "toxic" and two label it as "not toxic," standard practice applies a majority vote: the ground truth is set to "toxic," and the minority votes are discarded as errors.

However, for a wide class of natural language tasks, this objective ground truth assumption is fundamentally flawed. In tasks like emotion detection, offensiveness, framing, or political stance, disagreement is not an error—it is a valid reflection of differing human perspectives, values, and backgrounds. As Dallas Card et al. observe in their work on The Media Frames Corpus: Annotations of Frames Across Issues, variation in human labeling often captures legitimate alternative interpretations of complex semantic tasks.

When we force a consensus label onto inherently subjective text, we create an artificial performance ceiling (Ferrara et al.) and introduce rating indeterminacy, where multiple conflicting labels are equally valid (Guerdan et al.). To build models that truly understand human language, the field is transitioning from seeking a single objective truth to data perspectivism: an approach that explicitly embraces and models human label variation.

Why Aggregation Fails

Why does flattening annotator disagreement into a single majority label hurt models? Consider a sentence whose offensiveness depends heavily on cultural context. A majority vote silences minority perspectives, often erasing the voices of marginalized groups (Vitsakis et al.).

Furthermore, training models on aggregated labels introduces systematic flaws. In tasks like detecting online sexism or toxic spans, ambiguities in the dataset corrupt evaluation metrics and training signals if they are treated as binary choices (Verma et al., Kotyushev et al.). For example, Verma et al. highlighted that instances labeled as "not-sexist" often contain deep subjective ambiguities that standard metrics fail to capture.

By treating subjectivity as a moving target driven by individual beliefs (Yong et al.), perspectivist natural language processing (NLP) shifts the objective: instead of predicting the label, the model learns to predict who would assign which label.

Architectures for Modeling the Annotator

To mathematically represent multiple valid perspectives within a single model, researchers leverage two primary strategies: annotator-specific modeling and user personalization.

Annotator-Specific Heads

Instead of a single classification layer, models can be built with multiple specialized "heads," each dedicated to a specific annotator. In the ACTOR framework proposed by Xinpeng Wang et al. (ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation), a multi-head architecture learns individual annotator tendencies. This allows an active learning system to strategically query not just the most informative text sample, but the specific annotator whose perspective is most valuable to sample next. Similarly, Vitsakis et al. explore a multi-task setup to preserve individual annotator perspectives rather than defaulting to a majority baseline.

User Personalization

Another approach injects the observer's identity directly into the model's context. Joan Plepi et al. demonstrate that tracking individual annotator histories significantly improves predictions of social norm judgments on Reddit (Unifying Data Perspectivism and Personalization: An Application to Social Norms). For large language models, Wozniak et al. show that fine-tuning LLMs with simple User IDs boosts performance on subjective tasks like hate speech and emotion detection by up to 165% compared to non-personalized baselines.

Simulating Perspectives via Prompting

With the rise of generative LLMs, researchers have explored "sociodemographic prompting"—instructing a model to respond from the perspective of a specific demographic profile (e.g., age, race, or political leaning).

Zhou et al. show that injecting personal profiles into prompts allows LLMs to better simulate the subjective variance inherent in human cognitive appraisal. However, this approach comes with caveats. Beck et al. find that sociodemographic prompting can be highly fragile, where minor changes in prompt wording often outweigh the demographic profile itself. Furthermore, Orlikowski et al. demonstrate that LLMs are significantly better at memorizing the specific quirks of individual annotators than they are at learning the broader social patterns of demographic groups.

The Path Forward for Subjective Domains

Embracing data perspectivism changes how we collect data and evaluate models across diverse domains:

  • Finance: Determining whether a statement expresses "uncertainty" in financial disclosures is highly subjective; domain-specific embeddings can help capture this nuance (Theil et al.).
  • Politics: In political discourse, an observer's personal bias heavily dictates how they interpret speaker intent and conversational acts (Ferracane et al.).
  • News Consumption: To support this paradigm shift, new datasets are capturing rich annotator metadata alongside text. For instance, the iNews dataset introduced by Nigel Collier et al. pairs news articles with annotators' demographics, personality traits, and media trust metrics (iNews: A Multimodal Dataset for Modeling Personalized Affective Responses to News).

By moving away from the fiction of a single objective ground truth, these frameworks allow NLP models to navigate the rich, fragmented landscape of human perception.

Key Foundational Work

Word Embeddings-Based Uncertainty Detection in Financial Disclosures (2018) — contributed 1 concept elements to this direction.
The Media Frames Corpus: Annotations of Frames Across Issues (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subjective_label_ambiguity_f7ae2934,
  title = {subjective label ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-label-ambiguity/rdr_f7ae2934d26c4c2ab8e608d23d45d78d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_subjective_label_ambiguity_f7ae2934_v1,
  title = {subjective label ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-label-ambiguity/rdr_f7ae2934d26c4c2ab8e608d23d45d78d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake