ARTICLE: Annotator Reliability Through In-Context Learning (Student Abstract)

Sujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri, Christopher M. Homan, Ashiqur R. KhudaBukhshAAAI 2025 · DOI

Abstract

Subjective tasks like offensive speech detection often suffer from "noisy" labels, but traditional consensus-based filtering can silence valid minority perspectives. We present ARTICLE (Annotator Reliability Through In-Context Learning), a framework that identifies reliable annotators by measuring their internal consistency using Large Language Models (LLMs). By filtering for predictability rather than agreement, ARTICLE improves model performance while preserving demographic diversity in datasets.

Introduction

In machine learning, "ground truth" is typically derived from human annotations. However, for subjective tasks—such as determining if a text is "offensive"—disagreement is natural and often reflects diverse cultural or political backgrounds. Traditional quality control relies on outlier detection, which flags annotators who deviate from the majority. This approach risks creating "annotation echo chambers" by discarding legitimate minority viewpoints. The challenge is to distinguish between erratic, "noisy" annotators and those who hold consistent, albeit unique, perspectives.

Methodology: The ARTICLE Framework

The ARTICLE framework proposes a two-step approach to refine dataset quality:

  1. Consistency Filtering: Instead of comparing annotators to each other, ARTICLE compares an annotator to themselves. Using an LLM (Mistral-7B) in an In-Context Learning (ICL) setup, the model is given 10 of an annotator's previous labels and asked to predict their remaining labels. If the LLM can accurately model the annotator's logic (high F1-score), the annotator is deemed consistent. A hyperparameter $k$ defines the F1-score threshold for removal.
  2. Group-level Modeling: Once inconsistent annotators are removed, the remaining labels are aggregated by political affiliation (Democrat, Republican, Independent). The framework then models how these specific groups perceive offense using the cleaned data.

Experimental Results

The framework was evaluated on two offensive speech datasets: Toxicity Ratings ($D_{TR}$) and VOICED ($D_{VOICED}$). Results across both datasets and all political groups show a clear upward trend in model performance as the consistency threshold $k$ increases.

F1 scores across political groups for the first evaluation dataset as a function of the consistency threshold k.

As illustrated in the figures, a threshold of $k = 0.45$ consistently outperformed the baseline ($k = 0$), where no filtering was applied. This confirms that removing erratic signals allows models to better capture the nuanced logic of specific demographic groups.

F1 scores across political groups for the second evaluation dataset, demonstrating improved performance with higher consistency thresholds.

Discussion

The core insight of ARTICLE is that consistency is more valuable than consensus. By using an LLM's "ease of modeling" as a proxy for human reliability, the framework efficiently identifies high-quality signals without requiring multiple annotators per item. This method preserves the diversity of the dataset; for instance, it can retain both a Democrat and a Republican who disagree, provided both are internally logical.

However, there is a "Goldilocks Threshold." Setting $k$ too high (e.g., $> 0.5$) can lead to performance plateaus or declines, as it may remove too much data or filter out complex perspectives that are naturally harder to model but still valid.

Conclusion

ARTICLE demonstrates that LLMs can effectively filter human annotation noise by focusing on internal consistency. This approach not only improves model performance but also protects minority perspectives in subjective tasks. Future work will focus on automating threshold selection and applying the framework to a broader range of demographic lenses.

Summary updated

BibTeX
@inproceedings{dutta2025article,
  author = {Sujan Dutta and Deepak Pandita and Tharindu Cyril Weerasooriya and Marcos Zampieri and Christopher M. Homan and Ashiqur R. KhudaBukhsh},
  title = {{ARTICLE: Annotator Reliability Through In-Context Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35250},
}
Figure