Abstract
Natural Language Understanding (NLU) benchmarks rely heavily on crowdsourced data, but these datasets often suffer from repetitive linguistic biases known as annotation artifacts. While the community has traditionally blamed annotators for taking cognitive shortcuts, this paper reveals that these biases frequently originate from the dataset creators themselves—specifically from the limited examples provided in the annotation instructions. We demonstrate that annotators follow these instructions so closely that they replicate and even amplify the provided patterns. Our experiments show that models overfit to this "instruction bias," leading to inflated performance metrics and a failure to generalize to data that does not follow these dominant patterns.
Introduction
Progress in NLU is largely driven by benchmarks created via crowdsourcing platforms like Amazon Mechanical Turk. However, past research has identified "annotation artifacts"—unintended patterns or writing styles that allow models to cheat by learning shallow heuristics rather than actual language comprehension. Historically, the community has blamed annotators for these shortcuts. This paper challenges that assumption, asking: What if the annotators are just following instructions too well? The authors hypothesize that the few examples provided by dataset creators inadvertently teach crowdworkers to write highly repetitive, patterned data, leading to an "instruction bias" that infects the entire dataset.
Methods and Dataset Analysis
The authors conducted a multi-step analysis across 14 recent NLU benchmarks, including CLARIQ, COSMOSQA, DROP, DUORC, HOTPOTQA, HYBRIDQA, MC-TACO, MULTIRC, PIQA, QASC, QUOREF, ROPES, SCIQA, and WINOGRANDE. The methodology involved manually extracting dominant patterns from crowdsourcing instructions and tracking their propagation into the final datasets using string matching. Finally, they evaluated standard generative models (T5, BART) on subsets of the data to measure the impact of these patterns on generalization.
The Propagation of Instruction Bias
The analysis revealed a striking ubiquity of bias. On average, 72.7% of instruction examples share a single dominant pattern. This pattern propagates significantly, appearing in approximately 61% of the final collected dataset instances (59% in training sets and 62% in test sets).
In several datasets, such as QUOREF and ROPES, annotators actually amplified the bias, using the dominant pattern more frequently in the final dataset than it appeared in the original instructions. This suggests that the limited imagination of the prompt (averaging only 8.4 examples per task) constrains the diversity of the resulting data.
Impact on Model Performance
The presence of instruction bias creates an "illusion of competence." Because test sets contain the same biases as training sets, standard evaluation metrics artificially inflate model performance.
Catastrophic Generalization Failure
When models were trained only on instances containing instruction patterns ($S^P$), they suffered massive performance drops when evaluated on non-pattern instances ($S^{-P}$). As shown in the table below, the performance gap reached a staggering 58% in the DROP dataset, with significant drops exceeding 10% in CLARIQ, MULTIRC, PIQA, and QUOREF.
Performance Overestimation
Even when trained on the full dataset, models performed better on test examples containing the instruction pattern. Base models showed a ~7% performance gap, while larger models (like T5-large) were slightly more robust, showing a ~3% gap. This suggests that larger models' broader pre-training helps them generalize better, though they are still susceptible to the underlying bias.
Conclusion and Recommendations
The findings indicate that the way we instruct crowdworkers is a primary source of dataset bias. To mitigate this, the authors recommend providing a larger, more diverse pool of examples, potentially using LLMs for dynamic sampling. They also suggest active monitoring of word patterns during the data collection phase and explicitly evaluating models on pattern vs. non-pattern data splits to ensure true linguistic understanding.
Summary updated