Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models

Chaitanya Malaviya, Sudeep Bhatia, Mark Yatskar

Abstract

Human annotators often employ cognitive heuristics to minimize effort, creating systematic artifacts in datasets. This study investigates the link between annotator psychology and data quality in reading comprehension tasks. By introducing "annotator heuristic traces," we demonstrate that low-effort strategies correlate with psychological profiles and produce data that biased models exploit. These findings highlight a "cascading bias" where human shortcuts become model shortcuts, compromising generalization.

Introduction

NLP models are notorious "shortcut learners," often achieving high performance by exploiting spurious correlations rather than understanding language. While much research focuses on mitigating these artifacts in models, this paper investigates their origin in human behavior. We hypothesize that cognitive heuristics—mental shortcuts used to save time and effort—lead annotators to create predictable data artifacts that models subsequently learn to exploit.

Methodology: Measuring Heuristic Traces

To quantify low-effort strategies, the authors developed "annotator heuristic traces" categorized by psychological principles:

  • Satisficing: Measured via "low time" (completion speed) and "low effort" (keystroke-to-length ratio).
  • Availability Heuristic: Identified through "first option bias" (correct answer placement) and "serial position" (reliance on specific passage segments).
  • Representativeness: Tracked via "word overlap" and "copying" from the source text.

These traces were validated against the Cognitive Reflection Test (CRT), which assesses an individual's tendency to override intuitive, heuristic-driven responses with deliberative reasoning.

Correlation between psychological test scores (CRT) and heuristic traces.

The study found a significant correlation between an annotator's psychological profile and their data-writing behavior. Annotators who scored lower on the CRT were more likely to exhibit high heuristic traces in their work. The first principal component (PCA) of these traces proved to be the strongest predictor of an annotator's psychological profile.

Crucially, these human shortcuts "cascade" into the models. Biased models—such as "no-passage" baselines that should theoretically perform at chance—showed significantly higher precision on questions written by heuristic-heavy annotators. As the data pool moved from high-heuristic to low-heuristic examples, the performance of these biased models dropped sharply, indicating that the models were indeed latching onto human-generated shortcuts.

Precision of biased models across percentiles of heuristic traces, showing higher performance on heuristic-heavy data.

Impact on Data Quality and Generalization

Heuristic-seeking behavior fundamentally alters data quality. Questions from these annotators are more likely to be unsolvable, invalid, or solvable through simple word-matching rather than complex reasoning. This creates a generalization gap: models trained on heuristic-rich data perform significantly worse when evaluated on cleaner, more robust datasets. For instance, models trained on a "heuristic" split saw a performance drop of several percentage points compared to those trained on random splits.

Conclusion

The findings suggest that dataset artifacts are not random noise but systematic reflections of human cognitive shortcuts. To build more robust AI, researchers should monitor heuristic traces during the annotation process. Identifying these behaviors early allows for better worker training or the creation of more challenging datasets that force models to learn genuine linguistic patterns rather than human-provided "cheats."

Summary updated

Abstract

Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don't generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.
BibTeX
@inproceedings{malaviya2022cascading,
  author = {Chaitanya Malaviya and Sudeep Bhatia and Mark Yatskar},
  title = {{Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=-tFwTPai2O},
}
Figure