Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions

Mihir Parmar, Swaroop Mishra, Mor Geva, Chitta Baral

Abstract

Natural Language Understanding (NLU) benchmarks rely heavily on crowdsourced data, but these datasets often suffer from repetitive linguistic biases known as annotation artifacts. While the community has traditionally blamed annotators for taking cognitive shortcuts, this paper reveals that these biases frequently originate from the dataset creators themselves—specifically from the limited examples provided in the annotation instructions. We demonstrate that annotators follow these instructions so closely that they replicate and even amplify the provided patterns. Our experiments show that models overfit to this "instruction bias," leading to inflated performance metrics and a failure to generalize to data that does not follow these dominant patterns.

Introduction

Progress in NLU is largely driven by benchmarks created via crowdsourcing platforms like Amazon Mechanical Turk. However, past research has identified "annotation artifacts"—unintended patterns or writing styles that allow models to cheat by learning shallow heuristics rather than actual language comprehension. Historically, the community has blamed annotators for these shortcuts. This paper challenges that assumption, asking: What if the annotators are just following instructions too well? The authors hypothesize that the few examples provided by dataset creators inadvertently teach crowdworkers to write highly repetitive, patterned data, leading to an "instruction bias" that infects the entire dataset.

Methods and Dataset Analysis

The authors conducted a multi-step analysis across 14 recent NLU benchmarks, including CLARIQ, COSMOSQA, DROP, DUORC, HOTPOTQA, HYBRIDQA, MC-TACO, MULTIRC, PIQA, QASC, QUOREF, ROPES, SCIQA, and WINOGRANDE. The methodology involved manually extracting dominant patterns from crowdsourcing instructions and tracking their propagation into the final datasets using string matching. Finally, they evaluated standard generative models (T5, BART) on subsets of the data to measure the impact of these patterns on generalization.

The Propagation of Instruction Bias

The analysis revealed a striking ubiquity of bias. On average, 72.7% of instruction examples share a single dominant pattern. This pattern propagates significantly, appearing in approximately 61% of the final collected dataset instances (59% in training sets and 62% in test sets).

Dominant patterns identified in instructions and their propagation to training and test sets.

In several datasets, such as QUOREF and ROPES, annotators actually amplified the bias, using the dominant pattern more frequently in the final dataset than it appeared in the original instructions. This suggests that the limited imagination of the prompt (averaging only 8.4 examples per task) constrains the diversity of the resulting data.

Impact on Model Performance

The presence of instruction bias creates an "illusion of competence." Because test sets contain the same biases as training sets, standard evaluation metrics artificially inflate model performance.

Catastrophic Generalization Failure

When models were trained only on instances containing instruction patterns ($S^P$), they suffered massive performance drops when evaluated on non-pattern instances ($S^{-P}$). As shown in the table below, the performance gap reached a staggering 58% in the DROP dataset, with significant drops exceeding 10% in CLARIQ, MULTIRC, PIQA, and QUOREF.

Performance drop when models trained on pattern-heavy data are evaluated on non-pattern test sets.

Performance Overestimation

Even when trained on the full dataset, models performed better on test examples containing the instruction pattern. Base models showed a ~7% performance gap, while larger models (like T5-large) were slightly more robust, showing a ~3% gap. This suggests that larger models' broader pre-training helps them generalize better, though they are still susceptible to the underlying bias.

Conclusion and Recommendations

The findings indicate that the way we instruct crowdworkers is a primary source of dataset bias. To mitigate this, the authors recommend providing a larger, more diverse pool of examples, potentially using LLMs for dynamic sampling. They also suggest active monitoring of word patterns during the data collection phase and explicitly evaluating models on pattern vs. non-pattern data splits to ensure true linguistic understanding.

Summary updated

BibTeX
@inproceedings{parmar2023don,
  author = {Mihir Parmar and Swaroop Mishra and Mor Geva and Chitta Baral},
  title = {{Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions}},
  booktitle = {EACL},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.130},
}
Figure