ARTICLE: Annotator Reliability Through In-Context Learning

Sujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri, Christopher M. Homan, Ashiqur R. KhudaBukhshAAAI 2025 · DOI

TL;DR

This paper introduces ARTICLE, a novel framework that uses in-context learning with large language models (LLMs) to assess annotator reliability by measuring their self-consistency. By focusing on individual annotator patterns rather than just aggregate agreement, ARTICLE offers a more robust and potentially less resource-intensive way to ensure high-quality annotations, especially for subjective tasks.

What Problem Does This Paper Address?

Ensuring the quality of human annotations is crucial for training and evaluating machine learning models, particularly in Natural Language Processing (NLP). However, for subjective tasks like sentiment analysis or identifying offensive speech, disagreement among annotators can stem from genuine differences in opinion rather than poor quality work. Traditional methods often rely on outlier detection, which can suppress minority viewpoints and lead to "echo chambers." This paper tackles the challenge of reliably identifying annotator quality in a way that respects diverse perspectives and is efficient, especially when dealing with subjective data.

Methods

The core of the proposed framework, ARTICLE, is a two-step process leveraging in-context learning (ICL) with LLMs.

Step 1: Identifying Inconsistent Annotators * Individual LLM Modeling: Each annotator's labeling behavior is modeled individually using a high-performance LLM (Mistral-7B-instruct was primarily used). * Self-Consistency as a Proxy: The LLM is prompted to predict labels for a subset of an annotator's data, using another subset as in-context examples. The LLM's performance (measured by F1-score) on predicting these labels is used as a proxy for the annotator's consistency. A high F1 score suggests the annotator is easy to model and thus consistent; a low score indicates inconsistency. * Hyperparameter Threshold (k): A threshold k is introduced. Annotators whose F1 scores fall below k are flagged as inconsistent and removed.

Step 2: Modeling Group-Level Perception * Aggregate Labels: After removing inconsistent annotators, the remaining "consistent" annotators' labels are aggregated. * Group-Level ICL: The same LLM is then used with ICL to model the group-level perception of the task (e.g., offense) based on these aggregated labels.

This approach bypasses the need for extensive inter-annotator agreement metrics and aims to preserve unique, self-consistent viewpoints that might be discarded by traditional outlier detection.

Datasets

The framework was evaluated on two publicly available English offensive speech datasets:

  • Toxicity Ratings (DTR): A large dataset (107,620 comments) collected from platforms like Twitter, Reddit, and 4chan, annotated by 17,280 annotators. For experiments, a subset of 20,000 comments was used, ensuring each set of 20 comments was annotated by the same five annotators.
  • VOICED (DVOICED): A dataset of 2,338 YouTube comments from US cable news networks, annotated by 726 annotators.

Both datasets include annotators with diverse political leanings (Democrat, Republican, Independent), with at least 28% representation from each group. The original five-point toxicity scale was binarized into "offensive" and "non-offensive" categories.

Key Insights and Ideas

  • Self-Consistency as a Robust Quality Metric: The paper argues that an annotator's internal consistency, when evaluated by an LLM, is a more reliable indicator of quality than simply their agreement with the majority. This is particularly important for subjective tasks where genuine disagreement is common.
  • LLMs as "Judges" for Annotator Behavior: Leveraging LLMs not just for annotation but also for evaluating the quality of human annotations is a novel application. The LLM acts as a sophisticated judge of an annotator's consistency.
  • Preserving Minority Viewpoints: By focusing on self-consistency, ARTICLE can retain annotations from individuals who might hold minority opinions but express them consistently, thus avoiding the "echo chamber" effect of traditional methods.
  • Efficiency and Scalability: The ICL approach with LLMs is presented as potentially more resource-efficient than methods requiring large numbers of annotators for robust agreement metrics.

Key Results

  • Improved Modeling Performance: ARTICLE, particularly with a hyperparameter k set around 0.45, consistently outperformed the traditional CrowdTruth (CT) method in terms of F1-score for group-level modeling on both DTR and DVOICED datasets across all political groups.
  • Stability Across LLMs: The framework demonstrated stability when tested with different LLMs (Mistral-7B-instruct, Llama3-8B-instruct, and GPT-3.5-turbo), showing high Jaccard similarity in identifying inconsistent annotators.
  • Data Retention: At k = 0.45, ARTICLE retained a significant portion of annotators (around 70% in DTR, 55% in DVOICED) and comments (over 80% in both datasets), indicating it effectively filters out inconsistencies without discarding too much data.
  • Overlap with CT: While there was overlap between annotators identified as unreliable by ARTICLE and CT, a substantial number were flagged by only one method, suggesting they capture different aspects of annotation quality. ARTICLE's approach identified fewer annotators deemed unreliable by CT, and vice-versa.

Limitations and Future Work

  • Model Bias: The reliance on LLMs introduces potential biases inherent to these models, which could affect the accuracy of quality estimation, especially for nuanced linguistic or cultural contexts.
  • Handling Justified Disagreement: ARTICLE currently struggles to distinguish between genuine inconsistencies and justified disagreements arising from complex subjective interpretations.
  • Generalizability: The framework's effectiveness on other annotation tasks beyond offensive speech (e.g., medical imaging, legal documents) is yet to be verified.
  • Annotation Volume Dependency: The reliability of the self-consistency assessment is dependent on the volume of annotations provided by each annotator.
  • Future Work: The authors suggest exploring ways to combine ARTICLE with methods like CT to leverage their complementary strengths and further investigate its generalizability across diverse domains.

Summary updated

BibTeX
@inproceedings{dutta2025article,
  author = {Sujan Dutta and Deepak Pandita and Tharindu Cyril Weerasooriya and Marcos Zampieri and Christopher M. Homan and Ashiqur R. KhudaBukhsh},
  title = {{ARTICLE: Annotator Reliability Through In-Context Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i13.33558},
}
Figure