Building high-quality machine learning models requires massive volumes of labeled data. To scale up dataset creation, researchers and practitioners widely use pre-annotation: an existing model or heuristic tool generates initial labels, and human experts simply review and correct them. This human-in-the-loop pipeline promises the best of both worlds—the speed of automation combined with the accuracy of human judgment.
However, a robust body of evidence reveals a fundamental flaw in this assumption. Instead of acting as independent arbiters of truth, human annotators are deeply susceptible to automation bias and algorithmic anchoring. When presented with a pre-filled suggestion, the human brain shifts from a generative task (determining the correct label from scratch) to a verification task (evaluating if the proposed label is acceptable). This cognitive shift introduces systematic distortions that compromise the integrity of the resulting "gold standard" datasets.
The Cognitive Shift: From Generation to Verification
The root cause of pre-annotation bias lies in human cognitive heuristics. As demonstrated by Malaviya et al. in Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models, human annotators frequently employ mental shortcuts to minimize cognitive load and save time when performing repetitive tasks. When an automated system provides an initial suggestion, it acts as a psychological anchor.
Fort et al., in their foundational study Influence of Pre-Annotation on POS-Tagged Corpus Development, highlighted that while pre-annotation significantly accelerates workflows, it causes humans to subconsciously trust and accept incorrect machine suggestions. This cognitive inertia means that subtle or complex errors are routinely overlooked.
This phenomenon is especially pronounced in highly specialized domains where tasks are inherently ambiguous or difficult:
- Digital Pathology: Bertram et al. showed in Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels that incorporating algorithmic suggestions introduces confirmation bias, making human experts significantly more likely to agree with candidates proposed by a model than to identify them independently.
- Clinical NLP: Patel et al. observed in Annotation of a Large Clinical Entity Corpus that semi-automated tools introduce cognitive biases that lead medical experts to overlook errors in pre-filled text.
- Medical Imaging: Even when experts are explicitly tasked with refining complex spatial outputs, such as computer-aided tumor segmentations, Bereska et al. found in Development and external evaluation of a self-learning auto-segmentation model for Colorectal Cancer Liver Metastases Assessment (COALA) that the final annotations remain heavily anchored to the model's initial predictions.
- Ethnomusicology: In Automatic Music Transcription and Ethnomusicology: a User Study, Holzapfel et al. demonstrated that machine-generated drafts bias human transcribers toward the system's specific interpretations, reducing the diversity of human perspectives.
The Illusion of Consensus and Inflated Metrics
The danger of algorithmic anchoring is masked by two deceptive statistical artifacts: artificial inter-annotator agreement and inflated evaluation metrics.
Artificial Inter-Annotator Agreement
When multiple human annotators are given the same machine-generated starting point, they tend to overlook the same subtle errors and adopt the same systematic biases. As Fort et al. explain, this creates a false consistency where high inter-annotator agreement reflects the machine's systematic errors rather than objective ground truth. The baseline anchors the humans to the same conceptual space, hiding genuine labeling ambiguities.
Inflated Evaluation Metrics
This anchoring severely distorts how we evaluate machine learning models. In a seminal study, Berzak et al. demonstrated in Anchoring and Agreement in Syntactic Annotations that when a model is evaluated on a dataset that was pre-annotated by that same model (or a similar architecture) and subsequently corrected by a human, its performance metrics are artificially inflated. Because the human annotator subconsciously leaves the model's systematic errors intact, the model and the human share a "blind spot."
Berzak et al. found that this bias can artificially reduce observed error rates by 34% to 49% compared to evaluations conducted on data annotated entirely from scratch. Similar evaluation distortions have been documented across diverse tasks, from anatomical landmark localization (Tiulpin et al.) to evaluating large language model critics (the authors of CriticEval: Evaluating Large-scale Language Model as Critic).
[Raw Data] ──> [Pre-Annotation Model] ──> [Biased Suggestions]
│
▼ (Anchoring Bias)
[True Ground Truth] <── [Independent Human] [Human Validator]
│ │
▼ ▼
(Accurate Evaluation) (Inflated Metrics & Shared Blind Spots)
The Feedback Loop and Error Propagation
When datasets tainted by pre-annotation bias are used to train downstream models, they create a self-reinforcing feedback loop. Instead of learning the true underlying concepts, the new models learn to replicate the specific quirks, heuristics, and errors of the original pre-annotation tool.
In software engineering, Zhu et al. (An Efficient Design Smell Detection Approach with Inter-class Relation (S)) and Zhang et al. (Data Preparation for Deep Learning based Code Smell Detection: A Systematic Literature Review) note that training models on datasets labeled by automated tools causes the new models to inherit the rigid limitations of the original heuristics. Similarly, in crowdsourced web safety systems, Bhattacharya et al. showed in LookAhead: Augmenting Crowdsourced Website Reputation Systems With Predictive Modeling that showing automated predictions to human raters creates a feedback loop that entrenches model errors and prevents the acquisition of truly independent human consensus.
The rise of Large Language Models (LLMs) as annotation assistants has exacerbated this issue. When LLMs generate initial labels or reasoning chains, they introduce subtle cultural, linguistic, or stylistic predispositions that persist even after human validation, as documented by Ranjit et al. (OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants) and Yang et al. (SCCD: A Session-based Dataset for Chinese Cyberbullying Detection). As Gligoric et al. warn in Can Unconfident LLM Annotations Be Used for Confident Conclusions?, relying on unverified model outputs can lead to statistically confident but fundamentally incorrect scientific conclusions.
Breaking the Anchor: Mitigation Strategies
Recognizing that pre-annotation is necessary for scale but dangerous for quality, researchers have begun developing workflows designed to break the anchoring effect.
Multi-Agent Disagreement
One promising approach is to replace a single AI suggestion with a multi-agent framework. Min et al. demonstrated in Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages that exposing human annotators to a multi-agent debate system—where different models present conflicting perspectives—significantly reduces "label-following bias." By forcing the human to adjudicate between disagreeing AI agents rather than passively approving a single pre-filled label, the workflow restores critical evaluation and improves dataset reliability.
Confidence-Driven Routing
Instead of pre-filling every sample, frameworks like Confidence-Driven Inference, proposed by Gligoric et al., use model confidence scores to strategically route highly uncertain samples to humans for completely manual, from-scratch annotation. This preserves human cognitive resources for the cases where anchoring is most destructive, ensuring that the final dataset remains grounded in genuine human consensus.
Key Foundational Work
Influence of Pre-Annotation on POS-Tagged Corpus Development (2010) — contributed 4 concept elements to this direction.
Influence of Pre-annotation on POS-tagged Corpus Development (2012) — contributed 2 concept elements to this direction.
Discriminant Ranking for Efficient Treebanking (2010) — contributed 1 concept elements to this direction.