In machine learning, we treat "ground truth" as a north star. To find it, we typically ask several humans to label the same data and measure their Inter-Annotator Agreement (IAA). The logic seems sound: if three people independently agree that a sentence is "sarcastic" or a summary is "factually consistent," they must be right. High agreement becomes a proxy for high quality.
However, a growing body of research suggests that this consensus is often a mirage. High agreement does not guarantee that a label is correct; it only guarantees that the annotators are consistent. This distinction—between reliability (consistency) and validity (truth)—is the difference between a group of people all looking at the same map and a group of people all being in the right place.
Reliability is Not Validity
The fundamental problem is that a group of judges can be perfectly consistent while being perfectly wrong. Suzan Verberne and colleagues demonstrated this by comparing how external judges labeled search engine queries versus what the actual searchers intended. While the judges often agreed with each other (high reliability), they frequently failed to capture the user's actual goal (low validity).
This gap exists because external observers use shared external cues to make guesses, but those cues may not reflect the internal mental state of the person who created the data. As Mimansa Jaiswal and Rada Mihalcea found in emotion detection, consensus is often driven by simplified, shared biases rather than the nuanced context required to understand a speaker's true feelings.
The "Common Law" of Guidelines
Why do annotators agree so consistently on incorrect labels? Often, it is a byproduct of the annotation process itself. To increase agreement scores, researchers typically refine their guidelines, adding hyper-specific rules to handle edge cases.
Seunggun Lee and Nikita Nangia argue that this creates a “common law” within a research group. Annotators stop trying to interpret the data based on broad human concepts and instead follow a narrow, arbitrary logic designed specifically to boost statistical agreement. The resulting data is highly "reliable" within that group, but it loses its "validity" because it no longer represents how the broader population understands the concept.
Shared Ignorance and Systematic Bias
In technical or complex domains, high agreement often masks shared ignorance. If you ask three non-experts to label a complex biomedical sentence, they may all reach the same incorrect conclusion because they share the same lack of domain knowledge.
Yinfei Yang and Byron C Wallace showed that in specialized fields, agreement is a weak indicator of task difficulty. Similarly, Chenguang Wang and Alan Akbik found that even unanimous agreement among crowd workers can result in errors for complex linguistic tasks like Semantic Role Labeling.
This isn't just a problem of "bad workers." Lining Zhang and Elizabeth Clark observed that even "high-agreement" workers on platforms like MTurk often converge on superficial judgments rather than expert-level correctness. When a task is hard, humans—like models—look for shortcuts. If everyone takes the same shortcut, the agreement score looks great, but the data is flawed.
Toward Better Ground Truth
If simple consensus is a trap, how do we build better datasets? The research suggests several shifts in strategy:
- Task Routing: Instead of sending everything to the crowd, use models to predict "annotation difficulty." Yinfei Yang and Chenguang Wang both propose workflows where easy tasks go to the crowd and difficult ones are routed to experts.
- Beyond Majority Voting: Simple averaging or majority voting fails when tasks are subjective. Natalie Parde and Rodney D. Nielsen suggest using regression-based aggregation trained on a small set of expert-adjudicated data to predict the "true" label.
- Expert Intervention: Davide Bassi and Firoj Alam argue for probabilistic models that identify where annotators are likely to be systematically wrong, allowing for targeted expert intervention.
- Standardized Benchmarking: Projects like TRUE, led by Or Honovich and Roee Aharoni, aim to re-evaluate existing benchmarks to identify where human labels have failed, providing a cleaner foundation for detecting model hallucinations.
The takeaway is clear: we must stop treating agreement as the end of the conversation. High agreement is the beginning of a question: Are we agreeing because we found the truth, or because we've all learned to make the same mistake?