When building datasets for natural language processing, a low inter-annotator agreement (IAA) score is traditionally treated as a failure. Standard metrics like Cohen’s Kappa or Krippendorff’s Alpha collapse all annotator divergence into a single scalar value, framing every disagreement as random noise or human error.
However, a growing body of research argues that treating disagreement as a monolithic penalty obscures vital information. When two annotators disagree, they are often revealing something fundamental about the text: its structural complexity, its semantic ambiguity, or the presence of multiple valid interpretations. By moving beyond single-scalar metrics and systematically categorizing disagreements into structured taxonomies, researchers can diagnose flaws in annotation guidelines, target domain-specific knowledge gaps, and train models to handle real-world linguistic variation.
The Limits of the Single-Scalar Metric
To understand why decomposing disagreement works, consider how standard agreement metrics fail. If two annotators disagree on whether a sentence contains an argumentative claim, a metric like Kappa simply flags a mismatch. It cannot distinguish between a boundary dispute (where both agree a claim exists but disagree on its exact text span) and a fundamental perspective difference (where one annotator perceives a stance that the other does not).
As Anna Lindahl demonstrates in Disagreement in Argumentation Annotation, qualitative examinations of these discrepancies reveal that annotators frequently diverge due to structural boundaries or nuanced interpretations rather than outright mistakes. Similarly, Pihla Toivanen et al. show in Insights into developing analytical categorization schemes: three problem types related to annotation agreement that specific presentation styles—such as when information is negated, implicit, or highly sparse—systematically depress agreement scores in ways that standard metrics cannot diagnose.
By breaking down disagreements into explicit categories, researchers shift from asking whether annotators agree to understanding why they diverge.
Standard Approach:
[Annotator 1 Label] vs [Annotator 2 Label] ---> [ Kappa / Alpha Metric ] ---> Single Penalty Score
Taxonomic Approach:
---> Category A: Text Ambiguity (Keep as signal)
[Annotator 1 Label] vs [Annotator 2 Label] ---> Category B: Guideline Flaw (Fix instructions)
---> Category C: Annotator Error (Retrain human)
Decomposing the Roots of Divergence
When researchers begin to categorize disagreements, clear structural patterns emerge. One of the most intuitive ways to partition these errors is by separating the limitations of the human annotator from the inherent complexity of the text.
Semantic vs. Pragmatic Discrepancies
In specialized domains like clinical NLP, text interpretation requires both linguistic understanding and deep technical expertise. Taraka Rama et al. formalized this in Iterative development of family history annotation guidelines using a synthetic corpus of clinical text by introducing a semantic-pragmatic disagreement analysis:
- Semantic discrepancies stem from gaps in domain-specific knowledge (e.g., an annotator misinterpreting a complex medical condition).
- Pragmatic discrepancies arise from ambiguous or unclear procedural instructions in the annotation manual itself.
Distinguishing between these two types tells a project manager exactly how to fix their pipeline: semantic errors require more rigorous annotator training, whereas pragmatic errors require rewriting the technical guidelines.
Data-Centric vs. Annotator-Centric Taxonomies
This division between text properties and human properties scales well to broader tasks. In legal rationale extraction, Shanshan Xu et al. developed a two-level taxonomy in From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification that separates data-centric issues (such as missing context or text ambiguity) from annotator-centric issues (such as subjectivity or random noise).
When applied to multimodal tasks like Visual Question Answering (VQA), the taxonomy expands to accommodate the medium. Nilavra Bhattacharya et al. introduced a hierarchical taxonomy in Why Does a Visual Question Have Different Answers? that categorizes human disagreement into classes like input image quality, linguistic ambiguity, and semantic granularity. Across these domains, the intuition remains identical: treating label variation as a structured signal allows researchers to isolate human mistakes from legitimate text properties.
Algorithmic and Structural Discrepancy Mapping
For complex, hierarchical, or span-based tasks—such as parsing, named entity recognition (NER), or coreference resolution—disagreements cannot be categorized by simple label comparisons. Instead, researchers leverage structural algorithms to map out exactly where and how annotations diverge.
Derivation Trees and Structural Types
In syntactic treebank annotation, a single mistake can cascade through a phrase structure tree. To isolate the root causes, Seth Kulick et al. utilized derivation trees in Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation. By comparing fragments of these derivation trees, their system automatically groups inconsistencies by their precise structural type, pinpointing the exact grammatical constructions that cause annotator confusion.
Cluster-Based Discrepancy Analysis
For information extraction tasks where annotators highlight overlapping text spans, disagreements often manifest as complex "many-to-many" relationships. Bart Desmet et al. addressed this in QA4IE: A Quality Assurance Tool for Information Extraction by grouping overlapping annotations into spatial clusters.
Once clustered, the system automatically classifies the disagreements into explicit error types:
- Boundary shifts: Annotators agree on the entity but disagree on where it starts or ends.
- Split/combine issues: One annotator captures a phrase as a single span, while another breaks it into multiple sub-spans.
- Attribute mismatches: Annotators agree on the span but assign different categorical labels.
Annotator 1: [ Natural Language Processing ] (Technology)
Annotator 2: [ Natural Language ] (Linguistics) [ Processing ] (Computation)
Cluster Analysis ---> Identifies a "Split/Combine" structural mismatch rather than a binary error.
This structural decomposition is equally powerful for discourse analysis. In Disagreements in analyses of rhetorical text structure: A new dataset and first analyses, Freya Hewett et al. mapped hierarchical discourse trees to identify specific points of disagreement across categories like nuclearity, attachment points, and relation scope. Similarly, in cross-lingual settings, Ekaterina Lapshinova-Koltunski et al. developed a typology in Cross-lingual Incongruences in the Annotation of Coreference to classify coreference mismatches caused by translation shifts, such as explicitation (adding information during translation) or implicitation (omitting it).
Embracing Pluralism and Subjective Variance
Perhaps the most significant shift driven by disagreement categorization is the realization that some disagreements should never be resolved. In highly subjective tasks, variance in human labels is not noise—it is the ground truth.
Category-Stratified Agreement in Language Tasks
In Grammatical Error Correction (GEC), there are often dozens of valid ways to rewrite an ungrammatical sentence. Christopher Bryant et al. leveraged category-stratified agreement analysis in How Far are We from Fully Automatic High Quality Grammatical Error Correction? to decompose consensus based on error types. They discovered that mechanical tasks like subject-verb agreement yield high human consensus, whereas open-ended tasks like word choice feature massive, valid search spaces.
Subjective Preferences and Value Alignment
When evaluating the quality of AI-generated text, human preferences naturally diverge based on personal taste, background knowledge, and aesthetic standards. Jacopo Amidei et al. argue in Rethinking the Agreement in Human Evaluation Tasks that forcing annotators into perfect consensus in natural language generation tasks is counterproductive, as it stifles the inherent richness of human language.
To capture this systematically, Michael JQ Zhang et al. introduced a taxonomy of disagreement sources for LLM evaluation in Diverging Preferences: When do Annotators Disagree and do Models Know?. Their framework categorizes divergence into factors like task underspecification and preferred response style, demonstrating that over 30% of preference evaluation examples exhibit systematic, non-random divergence.
This perspective is critical when dealing with sensitive topics. As Anny D. Alvarez Nogales et al. show in Moral Disagreement over Serious Matters: Discovering the Knowledge Hidden in the Perspectives, texts with strong moral or political content naturally elicit disagreement based on the annotator's internal values. Rather than flattening these perspectives into a single majority vote, their work uses ensemble methods to preserve these diverse viewpoints, building models that understand the pluralistic nature of human values.
Conclusion
By treating human label variation as a structured signal rather than a binary error, disagreement categorization fundamentally changes how we build and evaluate NLP systems. Whether it is used to iteratively refine guidelines—as seen in the cross-lingual synchronization work by Kiki Ferawati et al. in Synchronizing Approach in Designing Annotation Guidelines for Multilingual Datasets: A COVID-19 Case Study Using English and Japanese Tweets—or to uncover hidden text ambiguities (as explored by Siyao Peng et al. in Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations), taxonomizing disagreement provides a diagnostic lens. It allows us to build datasets that reflect the true complexity of human language, ultimately leading to models that are better aligned with the nuanced, pluralistic world they are designed to navigate.
Key Foundational Work
Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation (2013) — contributed 2 concept elements to this direction.
ROSeAnn: Reconciling Opinions of Semantic Annotators (2013) — contributed 1 concept elements to this direction.
How Far are We from Fully Automatic High Quality Grammatical Error Correction? (2015) — contributed 1 concept elements to this direction.