Human Subjectivity and Disagreement in Semantic Annotation

42 concepts35 papers2 proposals

Supervised machine learning relies on a foundational assumption: given enough time and clear guidelines, human annotators can establish a definitive "ground truth." For straightforward tasks like part-of-speech tagging or named entity recognition, this assumption generally holds. However, as natural language processing expands into deeper semantic, pragmatic, and figurative layers of language, this consensus fractures.

When human experts—even native speakers—disagree on a label, it is rarely due to simple carelessness. Instead, it reflects the inherent subjectivity of language itself. Rather than treating inter-annotator disagreement as data noise to be engineered away, a growing body of research suggests that disagreement is a vital signal that reveals how humans navigate linguistic ambiguity.

Why Metaphors Defy Binary Classification

Metaphor detection serves as a primary battleground for annotation subjectivity. Is a phrase literal or figurative? The answer often depends on where it sits on a continuous spectrum of conventionalization.

Consider "dead" or conventionalized metaphors, such as "spending time" or "tackling a problem." As Alexandra Núñez et al. demonstrate in A "Wind of Change" - Shaping Public Opinion of the "Arab Spring" Using Metaphors, human annotators vary wildly in whether they perceive a highly common phrase as retaining its metaphorical quality or becoming purely literal. This variance introduces significant noise into gold-standard datasets.

The authors of VOLIMET: A Parallel Corpus of Literal and Metaphorical Verb-Object Pairs for English-German and English-French (Alexander Fraser et al.) reinforce this insight, noting that low inter-annotator agreement among experts suggests metaphoricity is fundamentally a spectrum rather than a binary choice. When forced into a binary box, fine-grained annotations become highly inconsistent, a problem highlighted by Yuri Bizzoni et al. in Bigrams and BiLSTMs Two Neural Networks for Sequential Metaphor Detection.

This challenge spans across domains and languages. For instance, Sebastian Reimann et al. highlight the inherent difficulty of metaphor annotation in specialized domains like online religious communication in Metaphors in Online Religious Communication: A Detailed Dataset and Cross-Genre Metaphor Detection. Even when cleaning benchmark datasets, Weicheng Ma et al. note in Improvements and Extensions on Metaphor Detection that the nuanced nature of metaphorical language presents a persistent challenge for evaluation.

The Problem of Internal Anchors and Context

Why exactly do annotators diverge? One major reason is the reliance on subjective "internal anchors." When evaluating the novelty of a metaphor, different people use different baselines for what feels creative versus mundane.

Erik-Lân Do Dinh et al. found in Weeding out Conventionalized Metaphors: A Corpus of Novel Metaphor Annotations that absolute rating scales fail because annotators lack a shared internal anchor for creativity. Similarly, Natalie Parde et al. show in Exploring the Terrain of Metaphor Novelty: A Regression-Based Approach for Automatically Scoring Metaphors that high human subjectivity makes establishing a reliable gold standard for linguistic shifts exceptionally difficult. When crowdsourcing these judgments, Sebastian Reimann et al. argue in When is a Metaphor Actually Novel? Annotating Metaphor Novelty in the Context of Automatic Metaphor Detection that ratings are often skewed by word rarity rather than genuine conceptual novelty, further complicating the baseline.

Furthermore, context shifts the boundary entirely. In A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry, Michael Toker et al. observe that an expression can be read as literal or metaphorical depending on the reader's understanding of the underlying mythological or poetic context. Even when evaluating proverbs, Gamze Goren et al. demonstrate in Context Matters: Enhancing Metaphor Recognition in Proverbs that explicit contextual cues fundamentally alter how abstract language is recognized. Without a shared, explicit context, annotators are left to fill in the blanks themselves, driving down agreement.

Subjectivity Across NLP Tasks

This challenge is not unique to metaphors; it spans many high-level NLP tasks where human judgment is central.

The Cost of Ignoring Disagreement

What happens when we ignore this disagreement and force a consensus? We introduce annotation bias.

Ehsan Aghazadeh et al. reveal the consequences of this bias in Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages. Because different datasets rely on different, unstandardized annotation criteria, models trained on one corpus suffer massive performance drops when tested on another. The models are not just learning the linguistic phenomenon; they are overfitting to the specific subjective biases of that dataset's annotators.

Cross-lingual settings compound this issue. Elisa Sanchez-Bayona shows in Meta4XNLI: A Cross-lingual Parallel Corpus for Metaphor Detection and Interpretation that inherent subjectivity persists across parallel corpora, requiring more annotation iterations to refine data quality. Meanwhile, the noise generated by these disagreements complicates downstream tasks like authorship attribution, as discussed by Gregorios A. Katsios et al. in Figuratively Speaking: Authorship Attribution via Multi-Task Figurative Language Modeling.

This issue persists even with rigorous guidelines. As Alec Sánchez-Montero et al. discover in Evaluating the Development of Linguistic Metaphor Annotation in Mexican Spanish Popular Science Tweets, even trained native speakers following strict procedures yield low inter-annotator agreement, proving that guidelines alone cannot completely erase human interpretive divergence.

Embracing the Spectrum

To build truly robust NLP systems, the field is shifting from eliminating disagreement to modeling it.

One approach is to capture continuous rather than binary relationships. For instance, in bridging reference resolution, Nobuhiro Ueda et al. demonstrate in Improving Bridging Reference Resolution using Continuous Essentiality from Crowdsourcing that crowdsourcing continuous scores of "essentiality" yields far more effective data than traditional, rigid labels.

Another path involves refining what we measure. Instead of annotating all metaphors, Stefanie Dipper et al. focus on intentionality in Guidelines for the Annotation of Deliberate Linguistic Metaphor, though they note that adding layers like novelty can make reliability assessment challenging.

Ultimately, the realization driving this research direction is that inter-annotator disagreement is not a bug to be fixed—it is a signal. By treating subjectivity as a core feature of language rather than data noise, models can better capture the rich, multi-layered ways humans actually communicate.

Key Foundational Work

Non-uniform Language Detection in Technical Writing (2016) — contributed 1 concept elements to this direction.
Mathematical Models for Balkan Phonological Convergence (1969) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_annotation_subjectivity_inter_annotator__30fd5e1c,
  title = {annotation subjectivity (inter-annotator agreement challenges)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-subjectivity-inter-annotator-agreement-challenges/rdr_30fd5e1cf7e94fffa8f9a37054519103},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_annotation_subjectivity_inter_annotator__30fd5e1c_v1,
  title = {annotation subjectivity (inter-annotator agreement challenges)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-subjectivity-inter-annotator-agreement-challenges/rdr_30fd5e1cf7e94fffa8f9a37054519103/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake