Abstract
Subjective tasks like offensive speech detection often suffer from "noisy" labels, but traditional consensus-based filtering can silence valid minority perspectives. We present ARTICLE (Annotator Reliability Through In-Context Learning), a framework that identifies reliable annotators by measuring their internal consistency using Large Language Models (LLMs). By filtering for predictability rather than agreement, ARTICLE improves model performance while preserving demographic diversity in datasets.
Introduction
In machine learning, "ground truth" is typically derived from human annotations. However, for subjective tasks—such as determining if a text is "offensive"—disagreement is natural and often reflects diverse cultural or political backgrounds. Traditional quality control relies on outlier detection, which flags annotators who deviate from the majority. This approach risks creating "annotation echo chambers" by discarding legitimate minority viewpoints. The challenge is to distinguish between erratic, "noisy" annotators and those who hold consistent, albeit unique, perspectives.
Methodology: The ARTICLE Framework
The ARTICLE framework proposes a two-step approach to refine dataset quality:
- Consistency Filtering: Instead of comparing annotators to each other, ARTICLE compares an annotator to themselves. Using an LLM (Mistral-7B) in an In-Context Learning (ICL) setup, the model is given 10 of an annotator's previous labels and asked to predict their remaining labels. If the LLM can accurately model the annotator's logic (high F1-score), the annotator is deemed consistent. A hyperparameter $k$ defines the F1-score threshold for removal.
- Group-level Modeling: Once inconsistent annotators are removed, the remaining labels are aggregated by political affiliation (Democrat, Republican, Independent). The framework then models how these specific groups perceive offense using the cleaned data.
Experimental Results
The framework was evaluated on two offensive speech datasets: Toxicity Ratings ($D_{TR}$) and VOICED ($D_{VOICED}$). Results across both datasets and all political groups show a clear upward trend in model performance as the consistency threshold $k$ increases.
As illustrated in the figures, a threshold of $k = 0.45$ consistently outperformed the baseline ($k = 0$), where no filtering was applied. This confirms that removing erratic signals allows models to better capture the nuanced logic of specific demographic groups.
Discussion
The core insight of ARTICLE is that consistency is more valuable than consensus. By using an LLM's "ease of modeling" as a proxy for human reliability, the framework efficiently identifies high-quality signals without requiring multiple annotators per item. This method preserves the diversity of the dataset; for instance, it can retain both a Democrat and a Republican who disagree, provided both are internally logical.
However, there is a "Goldilocks Threshold." Setting $k$ too high (e.g., $> 0.5$) can lead to performance plateaus or declines, as it may remove too much data or filter out complex perspectives that are naturally harder to model but still valid.
Conclusion
ARTICLE demonstrates that LLMs can effectively filter human annotation noise by focusing on internal consistency. This approach not only improves model performance but also protects minority perspectives in subjective tasks. Future work will focus on automating threshold selection and applying the framework to a broader range of demographic lenses.
Summary updated