Sparse Probability of Agreement

Jeppe Nørregaard, Leon DerczynskiarXiv 2022 · DOI · arXiv

Abstract

Inter-annotator agreement (IAA) is essential for validating ground truth in machine learning, but standard metrics like Fleiss’ Kappa require fully-annotated datasets. In practice, crowdsourced data is often sparse, with annotators labeling only subsets of items. This paper introduces the Sparse Probability of Agreement (SPA), a mathematically sound metric for incomplete data. We prove that SPA is an unbiased estimator of true agreement and evaluate weighting schemes to optimize performance. Our findings highlight the annotations_m1 scheme as a robust method for reducing variance in real-world applications.

Introduction

The reliability of human-labeled data is typically assessed through IAA metrics. However, classic approaches assume a "fully-annotated" set where every annotator labels every item. In large-scale crowdsourcing, this is rarely feasible due to annotator dropout or budget constraints. Researchers often resort to ad-hoc solutions that may introduce bias. There is a critical need for a metric that operates directly on sparse, incomplete datasets without sacrificing mathematical rigor.

Methodology: The SPA Metric

The authors propose Sparse Probability of Agreement (SPA), defined as a weighted micro-average of agreement across individual items.

Graph-Based Interpretation

A key contribution is modeling annotations as an undirected graph. In this model, nodes represent individual annotations, and edges represent agreement (the same category assigned to the same item). Agreement is calculated as the ratio of observed agreement edges to the total possible edges within the graph.

Graph-based interpretation of inter-annotator agreement where nodes are annotations and edges represent agreement.

Unbiased Estimation and Weighting

The paper proves that SPA remains an unbiased estimator of the "true" agreement even as annotations are randomly removed. To handle items with varying numbers of annotations ($n_i$), several weighting schemes were tested:

  • Flat: Every item counts equally.
  • annotations_m1: Items are weighted by $(n_i - 1)$, which naturally ignores items with only one label where agreement cannot be calculated.
  • Edge: Items are weighted quadratically by the number of possible agreement pairs.
  • Inv_var: A scheme that weights items by the inverse of their expected variance.

Experimental Evaluation

The metric was validated on four datasets: Discourse Acts on Reddit (DAR), Media Bias (MBIC), Brazilian Stock Market Tweets (BSMTWE), and Facial Expression Comparison (FECG). These datasets represent various levels of sparsity and category counts.

Key Results

Experiments confirmed that SPA is stable; as more annotations are added, the variance decreases while the estimate remains consistent. The annotations_m1 scheme emerged as a superior practical choice, offering a "Goldilocks" balance of simplicity and variance reduction.

Comparison of variance estimations (top) and the behavior of different weighting schemes (bottom) relative to the number of annotators.

As shown in the results, annotations_m1 and inverse-variance weighting consistently outperformed flat weighting in reducing variance, particularly in the MBIC and BSMTWE datasets. Unlike complex Bayesian models, SPA remains highly interpretable as the probability that two random annotators will agree on a random item.

Discussion and Future Work

SPA's primary advantage is its suitability for real-time monitoring during the annotation process. However, it currently measures observed agreement rather than chance-corrected agreement. Future research is needed to develop a "Sparse Kappa" that accounts for random agreement without introducing bias through statistical dependencies. Additionally, the metric assumes data is "Missing Completely at Random" (MCAR); systematic skipping of difficult items remains a challenge for future iterations.

Summary updated

Abstract

Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.
BibTeX
@misc{nrregaard2022sparse,
  author = {Jeppe Nørregaard and Leon Derczynski},
  title = {{Sparse Probability of Agreement}},
  year = {2022},
  eprint = {2208.06161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.06161},
}
Figure