Modeling Human Subjectivity in Natural Language Evaluation

295 concepts225 papers4 proposals

When evaluating whether an AI system has generated a "good" summary, a coherent essay, or an image that perfectly matches a prompt, we run into a fundamental obstacle: human judges rarely agree with one another.

Traditional machine learning evaluation relies on the assumption of a single, objective "gold standard" ground truth. However, in open-ended natural language generation (NLG) and understanding tasks, this assumption breaks down. Because different people prioritize different aspects of text quality—such as surface-level grammatical fluency versus deep narrative consistency—inter-annotator disagreement is an inherent property of the data, not just random noise.

To build reliable AI systems, researchers have had to move away from flattening human judgment into a single average score. Instead, they design evaluation frameworks that explicitly model, aggregate, or leverage this subjective variance.

The Failure of Absolute Metrics

Why do human evaluators disagree so frequently? One major driver is the cognitive limitation of absolute scoring systems. As Akash Kumar Mohankumar et al. observe in their work on Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons, asking humans to assign absolute scores to generated text introduces severe cognitive biases and sequence effects, where a judge's rating of a current text is heavily skewed by the quality of the text they read right before it.

Furthermore, broad evaluation instruments like Likert scales introduce significant variance. As Olivia Wiles et al. demonstrate in Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating, these scales suffer from low inter-annotator agreement because different judges interpret the boundaries of the scale differently.

Holistic Quality Score: [ 3 / 5 ]  <-- Highly unstable!

- Judge A focuses on grammar errors.
- Judge B focuses on logical contradictions.
- Judge C is influenced by the previous excellent example.

This disagreement is compounded by the fact that different readers naturally prioritize different types of errors. For instance, Tanya Goyal et al. show in SNaC: Coherence Error Detection for Narrative Summarization that assigning a single holistic score to long-form text is unreliable because judges weigh surface-level slips against deep logical contradictions entirely differently.

Quantifying Consensus via Atomic Units

If a single human reference cannot serve as an absolute ground truth, how can we build a stable benchmark? The key intuition is to shift the unit of evaluation from the entire text to fractional, atomic components of information.

By collecting multiple human-written references rather than just one, we can map out a distribution of what humans consider important. In their seminal paper on the Pyramid Method, Ani Nenkova et al. addressed content selection subjectivity by extracting discrete units of information from a pool of human summaries. Instead of treating one human's perspective as the truth, the authors weighted each information unit based on how many human annotators independently chose to include it.

[Factoid A] Mentioned by 5/5 humans (Weight: 5)
[Factoid B] Mentioned by 3/5 humans (Weight: 3)
[Factoid C] Mentioned by 1/5 humans (Weight: 1)

System Summary Score = Sum of weights of captured factoids

A system summary is then evaluated by how many of these highly-weighted, consensus-driven units it captures. Similarly, Simone Teufel et al. introduced a framework in Evaluating Information Content by Factoid Analysis: Human annotation and stability that breaks text down into discrete "factoids." This granular approach yields highly reproducible and stable system rankings, bypassing the volatile variance of individual holistic judgments.

Even automated metrics benefit from understanding this human variance. Chin-Yew Lin demonstrated in Looking for a Few Good Metrics: Automatic Summarization Evaluation - How Many Samples Are Enough? that while individual human judgments are highly subjective, aggregating these evaluations across a sufficient sample size allows automated metrics like ROUGE to achieve stable correlations with the collective human consensus. Furthermore, Karolina Owczarzak et al. showed that explicitly identifying and "cleaning" this assessor noise from training data directly improves the performance and reliability of automated scoring models.

Restructuring the Evaluation Task

Rather than trying to fix how humans assign scores, another highly effective strategy is to change the nature of the task we ask them to perform. Humans are notoriously bad at assigning absolute numbers, but they are highly consistent at making relative comparisons.

To exploit this, the framework proposed by Akash Kumar Mohankumar et al. uses a "dueling bandit" algorithm to evaluate NLG systems purely through pairwise comparisons (choosing whether Text A or Text B is better). This approach bypasses the scale-interpretation bias entirely, drastically reducing the human labor required to achieve a stable system ranking.

Alternatively, researchers can reduce subjectivity by narrowing the scope of the generation task itself. Emiel Krahmer et al. investigated this in Query-based Sentence Fusion is Better Defined and Leads to More Preferred Results than Generic Sentence Fusion. They discovered that generic text-merging tasks suffer from low agreement because there are simply too many valid ways to combine information. By introducing a specific query to guide the task, they constrained the evaluation context, causing human consensus to rise naturally.

Embracing Multi-Faceted Perspectives

The most recent shift in this research direction abandons the pursuit of a single consensus altogether. Instead of viewing inter-annotator disagreement as a problem to be solved, it treats subjectivity as a feature of a diverse user base.

A single summary might be perfect for a casual reader but entirely inadequate for a technical expert. To capture this reality, Ning Wu et al. introduced the Diverse Role-Player Evaluation (DRPE) framework in Large Language Models are Diverse Role-Players for Summarization Evaluation. Instead of forcing an evaluation into a single metric, the authors prompt Large Language Models to simulate a variety of distinct human stakeholders—ranging from professional critics to specific end-users.

By explicitly modeling these diverse personas, the evaluation framework embraces the multi-faceted nature of human subjectivity, providing a richer, more diagnostic picture of system performance than any single gold standard ever could.

Key Foundational Work

Evaluation of Semantic Clusters (1995) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inter_annotator_disagreement_f55998aa,
  title = {inter-annotator disagreement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inter-annotator-disagreement/rdr_f55998aae24e4fab9ffbb4579edbd58f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_inter_annotator_disagreement_f55998aa_v1,
  title = {inter-annotator disagreement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inter-annotator-disagreement/rdr_f55998aae24e4fab9ffbb4579edbd58f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake