Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models

Chaitanya Malaviya, Sudeep Bhatia, Mark Yatskar

Abstract

Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don't generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.

Introduction

The "garbage in, garbage out" principle is a fundamental tenet of machine learning, yet the specific nature of that "garbage" is often poorly understood. In Natural Language Processing (NLP), crowdsourced datasets frequently contain "artifacts"—unintended patterns or shortcuts that allow models to achieve high scores without actually learning the underlying task. For instance, a model might learn that the presence of the word "not" in a premise always correlates with a specific label in a sentiment task.

This research argues that these artifacts are not random noise. Instead, they are the systematic result of human psychology. Annotators, acting as "cognitive misers," utilize heuristics to minimize mental effort and time. This paper bridges the gap between cognitive psychology and data engineering by measuring these shortcuts and demonstrating how they directly degrade the robustness and generalization of machine learning models.

Measuring the "Cognitive Miser": Methods

To quantify these psychological shortcuts, the researchers developed a framework called Annotator Heuristic Traces. Rather than evaluating only the final labels, this approach tracks the process and characteristics of the data generation itself.

1. Feature Tracking

The study identified several "traces" indicative of low-effort strategies:

  • Satisficing: This involves spending the minimum amount of time necessary to complete a task. Traces include very low time-per-task or unusually short question lengths.
  • Availability Bias: This manifests as a tendency to rely on the most easily accessible information. For example, annotators might disproportionately make the first provided option the correct answer, or ensure the answer is always located in the first or last sentence of a passage.
  • Representativeness: This is often seen as "word-matching." The researchers measured word overlap between questions and passages and used Longest Common Subsequence (LCS) algorithms to identify copy-pasting behavior.

2. Psychological Validation

To confirm that these traces reflect actual cognitive tendencies, annotators completed the Cognitive Reflection Test (CRT). The CRT consists of "trick" questions where an intuitive but incorrect answer must be overridden by conscious reflection. This allowed the researchers to correlate data quality with the annotators' innate tendency to use heuristics.

3. Biased Model Probing

The researchers tested the resulting data against "partial input models." These are models intentionally deprived of context (e.g., a model that sees the question but not the passage). If a partial input model can solve a task, it indicates the data contains exploitable shortcuts rather than requiring true comprehension.

Experimental Results

The study utilized a primary dataset of 1,225 multiple-choice reading comprehension pairs from 73 annotators, supplemented by validation on external datasets from Sugawara et al. (2022).

The findings revealed a significant correlation between an annotator’s CRT performance and the heuristic traces in their work. Annotators who failed to reflect on logic puzzles were the same individuals who produced "lazier" data characterized by high lexical overlap and predictable answer positioning.

The Impact on Model Robustness

  • Exploitation by Biased Models: Examples created by heuristic-seeking annotators were significantly easier for biased models to solve. A simple "Lexical Overlap" model performed substantially better on data from the top 25% of heuristic-seeking annotators compared to the rest of the pool.
  • Generalization Failure: Models trained on data from "lazy" annotators generalized poorly to new contexts. When a model is trained on data that relies on word-matching, the model itself becomes a word-matcher. Consequently, these models showed a performance drop of 3-4% in accuracy when tested on more complex, non-heuristic data.
  • Qualitative Decline: Manual audits showed that heuristic-heavy annotators were more likely to produce unsolvable or ambiguous questions. Their valid questions typically required simple pattern matching rather than the multi-sentence reasoning the task was designed to evaluate.

Conclusion and Future Directions

The study demonstrates a "cascading bias": human cognitive shortcuts create artifacts in data, which in turn produce brittle models that fail in real-world scenarios. While the study focused on reading comprehension, the implications extend to any task where human effort is a variable.

The authors suggest that dataset creators should move beyond simple quality checks and begin tracking "heuristic traces" in real-time. By identifying these patterns during the collection phase, researchers can provide immediate feedback to annotators or filter out low-quality data before it compromises model training. Future work will explore how these heuristics manifest in other domains, such as translation and image labeling, and whether factors like fatigue or task difficulty can predict when an annotator will begin to rely on shortcuts.

Summary updated

Abstract

Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don't generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.
BibTeX
@inproceedings{malaviya2022cascading,
  author = {Chaitanya Malaviya and Sudeep Bhatia and Mark Yatskar},
  title = {{Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=-tFwTPai2O},
}

Loading related content...

Figure