Sequence Effects in Crowdsourced Annotations

Nitika Mathur, Timothy Baldwin, Trevor Cohn

Introduction

In the field of Natural Language Processing (NLP), crowdsourced annotations are frequently treated as the "ground truth" for training and evaluating state-of-the-art models. To ensure fairness and reduce bias, researchers typically randomize the order of items within a task (often called a Human Intelligence Task or HIT). However, a common practice is to present that same randomized sequence to every worker who attempts the task.

This paper argues that such a practice creates a systematic "sequence effect." Human judgment is inherently relative; our perception of an item is influenced by the context of what we have just seen. If every worker follows the same sequence, individual psychological biases—such as anchoring or the gambler’s fallacy—do not cancel out. Instead, they aggregate into a systematic bias that can distort model evaluations and leaderboard rankings.

Quantifying the Sequence Effect

To detect these biases, the researchers utilized a linear regression model designed to hunt for autocorrelation—the statistical relationship between a current annotation and the one immediately preceding it. The model is defined as:

$$Y_{i,t} = \beta_0 + \beta_1 Y_{i,t-1} + \beta_2 Gold + \eta$$

In this equation, $Y_{i,t}$ represents the current annotation, $Y_{i,t-1}$ is the previous annotation, and $Gold$ is the expert label used to control for the actual quality of the item. The coefficient $\beta_1$ is the primary metric of interest: a positive $\beta_1$ suggests anchoring or assimilation (giving a score similar to the last one), while a negative $\beta_1$ suggests contrast effects or the gambler’s fallacy (a tendency to switch labels or "correct" a perceived streak).

Binary Tasks and the Gambler’s Fallacy

In binary classification tasks, such as Recognizing Textual Entailment (RTE) and event ordering (TEMPORAL), the study found significant negative autocorrelation. This behavior aligns with the "Gambler’s Fallacy": if a worker has seen several "True" labels in a row, they subconsciously feel a "False" label is "due," regardless of the actual content of the next item.

Autocorrelation coefficients for binary tasks.

As shown in the data, these effects are particularly strong among moderate-quality workers, with coefficients reaching as low as $-0.511$ in the TEMPORAL task. This indicates a strong psychological pressure to alternate labels when the ground truth is perceived as repetitive.

Continuous Scales and Anchoring

Conversely, tasks involving continuous scales—such as judging Machine Translation (MT) adequacy on a 100-point sliding scale—exhibited positive autocorrelation. In these scenarios, workers tend to use their previous score as an "anchor," adjusting only slightly for the next item rather than treating it as a fresh judgment.

Sequence effects in MT Adequacy by worker quality.

The study also revealed that worker quality plays a significant role in the magnitude of this bias. While "good" workers (those whose labels correlate well with experts) still showed a significant bias of $\beta_1 = 0.030$, "bad" workers were far more susceptible, showing a much higher sequence bias of $\beta_1 = 0.192$.

The Calibration Phase and Systemic Impact

The researchers observed that sequence effects are most pronounced at the beginning of a task. As workers annotate more items, they undergo a "calibration phase" where they refine their internal scales. For instance, the bias coefficient dropped from $0.044$ in the first tertile of a task to $0.015$ by the third.

Despite this calibration, the aggregate impact remains severe. In the MT dataset, a translation following a "low quality" item received a significantly lower score than the same translation following a "high quality" item. This gap was measured at $0.18$ standardized units ($p < 0.001$), a difference large enough to potentially flip the rankings of competing MT systems in a formal evaluation.

Conclusion and Recommendations

The primary takeaway for researchers is that standard HIT-level randomization is insufficient. To prevent psychological quirks from baking systematic bias into datasets, the authors recommend per-worker randomization. By ensuring every worker sees a unique permutation of items, systematic sequence effects are converted into random noise, which can then be safely mitigated through standard averaging techniques.

Summary updated

BibTeX
@inproceedings{mathur2017sequence,
  author = {Nitika Mathur and Timothy Baldwin and Trevor Cohn},
  title = {{Sequence Effects in Crowdsourced Annotations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1306},
}
Figure