Filtering Training Data via Annotator Consensus

74 concepts70 papers2 proposals

Supervised machine learning relies heavily on labeled data, but human annotations are notoriously noisy. Whether due to the inherent subjectivity of tasks like emotion recognition or the varying expertise of crowdsourced workers, conflicting labels introduce a destructive fog into the training signal. Instead of treating all collected data equally or relying on complex generative models to guess the "true" label, a remarkably effective paradigm has emerged: consensus-based label filtering.

The core intuition is simple: by treating multi-annotator agreement as a proxy for data quality, we can prune ambiguous or erroneous examples entirely. This leaves a high-purity subset that provides a significantly clearer training signal, proving that sometimes, less data of higher quality is exactly what a model needs to excel.

The Power of Purity Over Volume

Traditional deep learning wisdom suggests that more data is always better. However, consensus filtering challenges this assumption by prioritizing the purity of the training signal over raw volume. When human annotators disagree, the resulting label noise acts as entropy that degrades model performance.

As shown by Yong et al., training models on a smaller subset of data where human annotators reach a unanimous consensus can yield better accuracy than using larger datasets based on simple majority-rule labels. This phenomenon is particularly pronounced in highly subjective or nuanced domains. For instance, in the syndromic classification of social media messages, Doan demonstrated that restricting the training set to examples with unanimous agreement ensures a clean signal capable of distinguishing subtle contextual differences.

Similarly, Kowtha et al. found that the degree of human annotator consensus directly predicts the performance of speech emotion recognition models, confirming that low-consensus samples primarily function as noise that hinders optimization.

Synthesizing Expertise from the Crowd

Why does filtering based on non-expert consensus work so well? The answer lies in the statistical cancellation of individual errors. While an individual non-expert worker may introduce random errors or misinterpret instructions, aggregating independent judgments allows a clear, reliable signal to emerge.

In a foundational study, Snow et al. demonstrated that aggregating a small number of independent non-expert judgments can produce data quality comparable to that of a single domain expert. This collective intelligence scales remarkably well across complex domains:

  • Speech and Language: Marge et al. showed that a unanimous consensus among non-expert annotators serves as a highly reliable proxy for expert-level judgment in subjective speech summarization tasks.
  • Medical Imaging: Duggan et al. revealed that aggregating as few as seven quality-filtered opinions from non-experts via a gamified platform yields classification accuracy on lung ultrasound data that is statistically indistinguishable from fellowship-trained experts.

By filtering out instances that fail to reach a high consensus threshold, researchers can rapidly scale dataset annotation using non-experts without sacrificing the integrity of the ground truth.

Mechanisms of Consensus Filtering

Consensus filtering can be implemented at various levels of strictness depending on the desired balance between precision and recall:

Unanimous Filtering

The strictest approach dictates that an instance is included only if all independent annotators assign the exact same label. This is often used to build high-precision evaluation benchmarks or clean seeds for bootstrapping, as seen in Mandarin polyphone disambiguation by Park and Romanian word sense disambiguation by Mihalcea et al..

Threshold-Based Pruning

Instances are retained if the agreement ratio exceeds a specific mathematical threshold (such as a two-thirds majority). This allows researchers to systematically tune the trade-off between data volume and label purity, a technique explored by Recht et al. to evaluate how different levels of data filtering affect the measured accuracy of machine learning models on ImageNet.

Multi-System Voting

The concept extends beyond human annotators to automated systems. When generating "silver-standard" datasets from multiple legacy models or large language models (LLMs), requiring a multi-system voting threshold effectively filters out system-specific hallucinations and idiosyncratic errors, as detailed by Lewin et al. and Chen et al..

Filtering the Annotators

Rather than filtering data purely post-hoc, advanced consensus frameworks optimize the process by dynamically filtering the annotators themselves. By evaluating crowd workers against known gold-standard examples or tracking their agreement relative to their peers, low-performing contributors or spammers can be systematically pruned from the pool.

Endriss et al. proposed an iterative aggregation strategy that fixes labels for high-consensus items first, using those established values to calculate reliability scores and filter out sub-threshold contributors. Similarly, Zhang et al. demonstrated that filtering out novice annotators within specific data clusters ensures high accuracy even when the overall majority of the crowd is unreliable.

Ultimately, consensus-based filtering works because it treats disagreement not as a failure of the annotation process, but as a valuable diagnostic signal. By systematically removing the friction of ambiguous data, it allows machine learning models to learn the core underlying patterns with far greater efficiency.

Key Foundational Work

An evaluation exercise for Romanian Word Sense Disambiguation (2004) — contributed 2 concept elements to this direction.
The Senseval-3 Multilingual English-­Hindi lexical sample task (2004) — contributed 1 concept elements to this direction.
Using Readers to Identify Lexical Cohesive Structures in Texts (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_consensus_based_label_filtering_ff3d8222,
  title = {consensus-based label filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/consensus-based-label-filtering/rdr_ff3d822214ca40219b1f8aa472ceaf02},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_consensus_based_label_filtering_ff3d8222_v1,
  title = {consensus-based label filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/consensus-based-label-filtering/rdr_ff3d822214ca40219b1f8aa472ceaf02/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake