Introduction
Machine learning models for sentiment analysis are only as good as the data they are trained on. However, generating high-quality "ground truth" labels through human annotation is notoriously difficult. This paper addresses anchoring bias: the psychological phenomenon where an annotator’s first few interactions with a dataset create a biased reference point. If a worker starts their task by seeing several extremely negative reviews, they might perceive a subsequent "moderately negative" review as neutral by comparison. This sequential bias leads to inconsistent labeling and higher Mean Absolute Error (MAE).
The Persistence of Bias
A key finding of this research is that bias is not a temporary "warm-up" issue that resolves itself. Without intervention, annotator error actually increases during the initial phase of a task and fails to stabilize quickly.
As seen in Figure 2, error peaks early in the sequence. While existing solutions often rely on redundant labeling or "gold standard" trap questions, these methods are expensive and labor-intensive. The authors instead propose an Intelligent Calibration phase—a small set of $k$ reviews designed to stabilize judgment before the actual task begins.
Methodology
To find the optimal calibration set, the authors modeled bias using logistic regression and derived five specific rules for the sequence:
- Rules 1 & 2: Avoid starting with extremely positive or extremely negative reviews.
- Rule 3: The second review should be exceptionally negative (bottom 20th percentile).
- Rule 4: The third review should be neutral (ground truth around 3.0).
- Rule 5: Maintain low standard deviation for the second and third reviews.
The team validated these rules through 17 experiments involving 1,640 crowdworkers across Prolific and Amazon Mechanical Turk.
Key Results
The calibrated approach significantly outperformed both non-calibrated and randomly calibrated groups, achieving an MAE of 0.47 compared to 0.58 for the non-calibrated group.
As illustrated in Figure 3, the "Calibration" group (green) consistently maintains the lowest error throughout the task. In the food domain, this led to a 19% reduction in MAE. Remarkably, when these rules were applied to hotel reviews, the improvement reached 24%, suggesting that anchoring bias is a fundamental human cognitive trait rather than a domain-specific quirk.
Conclusion
Intelligent calibration offers a high-efficiency, low-overhead method for improving data quality. By investing a few minutes in a specific "Aha" sequence at the start of a task, researchers can achieve levels of accuracy in the first five reviews that uncalibrated workers may never reach. This "fixed-cost" overhead is far more efficient than majority-voting and ensures more consistent sentiment corpora.
Summary updated