Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts

Georgios Chochlakis, Peter Wu, Arjun Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan

Abstract

Modeling complex subjective tasks in Natural Language Processing, such as recognizing emotion and morality, is considerably challenging due to significant variation in human annotations. This variation often reflects reasonable differences in semantic interpretations rather than mere noise, necessitating methods to distinguish between legitimate subjectivity and error. We address this challenge by exploring label verification in these contexts using Large Language Models (LLMs). First, we propose a simple In-Context Learning binary filtering baseline that estimates the reasonableness of a document-label pair. We then introduce the Label-in-a-Haystack setting: the query and its label(s) are included in the demonstrations shown to LLMs, which are prompted to predict the label(s) again, while receiving task-specific instructions (e.g., emotion recognition) rather than label copying. We show how the failure to copy the label(s) to the output of the LLM are task-relevant and informative. Building on this, we propose the Label-in-a-Haystack Rectification (LiaHR) framework for subjective label correction: when the model outputs diverge from the reference gold labels, we assign the generated labels to the example instead of discarding it. This approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. Comprehensive analyses, human evaluations, and ecological validity studies verify the utility of LiaHR for label correction.

Introduction

In "objective" NLP tasks like mathematical reasoning or fact-checking, annotator disagreement is typically a clear indicator of error. However, in "subjective" domains—such as identifying emotions, moral foundations, or harmful speech—disagreement often reflects valid differences in perspective rooted in cultural background or personal experience. The central challenge for researchers is distinguishing between legitimate subjectivity (diverse but valid opinions) and actual errors (human "hallucinations" or noise).

Traditional error-correction methods often rely on majority voting, which risks erasing minority perspectives and flattening the rich diversity of human interpretation. This paper introduces a novel approach to clean noisy datasets while preserving valid subjectivity by leveraging the internal priors of Large Language Models (LLMs).

Methodology: Label-in-a-Haystack Rectification (LiaHR)

The core innovation of this work is the Label-in-a-Haystack Rectification (LiaHR) framework. In a standard few-shot In-Context Learning (ICL) setting, models are shown several demonstrations of a task. In the LiaHR setting, the specific test query and its provided "gold" label are included as one of these demonstrations. Crucially, the LLM is instructed to perform the task (e.g., "Classify the emotion") rather than simply copy the labels.

The Label-in-a-Haystack Rectification (LiaHR) process. By embedding the target input and label within the prompt's demonstrations, researchers use the model's internal priors to verify human annotations.

If the LLM's internal knowledge (its "prior") strongly disagrees with the provided label, it will fail to copy the label and instead generate what it considers a more plausible alternative. This "failure" to follow the prompt's example serves as a high-precision signal that the human label might be an error. When the model outputs diverge from the reference gold labels, the framework assigns the generated labels to the example instead of discarding it.

Experimental Setup and Proxy Properties

The authors evaluate LiaHR across several benchmarks known for high subjectivity, including SemEval 2018 Task 1 E-c (emotion recognition in tweets), GoEmotions (Reddit comments), and the Moral Foundations Reddit Corpus (MFRC). They also use QueerReclaimLex to test how models handle reclaimed slurs based on identity.

To assess performance, the study defines four "proxy properties":

  1. Nonconformity: The ability to not blindly follow the prompt when the label is wrong.
  2. Diversity: The capacity to accept multiple valid labels for a single text.
  3. Noise Rejection: The ability to flag randomly shuffled labels.
  4. Rectification: The quality of the proposed alternative labels.

Key Results

Model Performance and Noise Rejection

Capable models like GPT-4o and Llama-3 70B successfully rejected random/shuffled labels at much higher rates than they rejected actual gold labels. This demonstrates that the models are not disagreeing at random but are responding to the semantic quality of the annotation.

Label-in-a-Haystack success rates on SemEval 2018 Task 1. Capable models like GPT-4o and Llama-3-70b show high conformity to gold labels while rejecting random ones, demonstrating their utility in error detection.

Human Preference and Downstream Utility

In human evaluations, when LiaHR disagreed with the original human "gold" label, human judges preferred the LLM’s corrected label a significant portion of the time. For example, using GPT-4o on the SemEval dataset, humans preferred the model's correction in 49 cases compared to only 16 for the original gold label.

Furthermore, the authors "rectified" training sets using LiaHR and trained smaller BERT-based models on the cleaned data. On SemEval, the Micro F1 score improved from 0.689 (original) to 0.692 (rectified), proving that the corrected labels provide a cleaner signal for supervised learning.

Discussion

The U-Shaped Trend in Model Scale

The authors hypothesize a U-shaped relationship between model size and performance in the LiaHR task. Very small models lack the instruction-following capability to conform even to correct labels. Mid-sized models often follow instructions too well, blindly copying errors. The largest, most capable models follow instructions but allow their strong internal priors to override unreasonable inputs, making them the best candidates for label verification.

Limitations and Ethical Considerations

LLMs are not perfect oracles and carry their own systemic biases. On the QueerReclaimLex dataset, models were slightly less accurate at identifying in-group reclaimed speech, suggesting their sense of "reasonableness" is filtered through training data biases. Consequently, LiaHR is best viewed as a tool within an annotation pipeline—flagging suspicious labels for secondary human review—rather than a fully automated replacement for human judgment.

Conclusion

LiaHR offers a clever reversal of the standard prompting paradigm: by treating a model's failure to copy a label as a feature rather than a bug, researchers can identify human errors in complex, subjective datasets. This framework provides a scalable way to enhance the signal-to-noise ratio in NLP benchmarks while respecting the inherent diversity of human perspectives.

Summary updated

Abstract

Modeling complex subjective tasks in Natural Language Processing, such as recognizing emotion and morality, is considerably challenging due to significant variation in human annotations. This variation often reflects reasonable differences in semantic interpretations rather than mere noise, necessitating methods to distinguish between legitimate subjectivity and error. We address this challenge by exploring label verification in these contexts using Large Language Models (LLMs). First, we propose a simple In-Context Learning binary filtering baseline that estimates the reasonableness of a document-label pair. We then introduce the Label-in-a-Haystack setting: the query and its label(s) are included in the demonstrations shown to LLMs, which are prompted to predict the label(s) again, while receiving task-specific instructions (e.g., emotion recognition) rather than label copying. We show how the failure to copy the label(s) to the output of the LLM are task-relevant and informative. Building on this, we propose the Label-in-a-Haystack Rectification (LiaHR) framework for subjective label correction: when the model outputs diverge from the reference gold labels, we assign the generated labels to the example instead of discarding it. This approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. Comprehensive analyses, human evaluations, and ecological validity studies verify the utility of LiaHR for label correction. Code is available at https://github.com/gchochla/LiaHR.
BibTeX
@inproceedings{chochlakis2025humans,
  author = {Georgios Chochlakis and Peter Wu and Arjun Bedi and Marcus Ma and Kristina Lerman and Shrikanth Narayanan},
  title = {{Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure