Learning from Noisy Crowdsourced Annotations

605 concepts517 papers4 proposals

Modern machine learning relies heavily on massive labeled datasets. To scale up data collection, researchers frequently turn to crowdsourcing platforms like Amazon Mechanical Turk. However, shifting from a small group of domain experts to a large crowd of non-experts introduces a fundamental challenge: label noise.

Crowdsourced annotations are rarely perfectly clean. Workers possess varying levels of expertise, operate under different subjective biases, and may exhibit varying levels of attentiveness or effort. Traditional supervised learning algorithms assume a single, pristine "gold standard" ground truth. When trained directly on raw, noisy crowd labels, these models often overfit to human errors and systematic biases, degrading their ability to generalize.

To solve this, a major research direction focuses on designing algorithms that can robustly learn the true underlying signal from a collection of noisy, conflicting, and incomplete annotations.

The Failure of Simple Majority Voting

The naive approach to handling multiple crowd annotations per instance is Majority Voting (MV): simply taking the label chosen by the majority of workers and treating it as the ground truth. While intuitive, MV fails because it treats all annotators and all data instances as equal.

In reality, crowdsourced noise is highly structured:

  1. Annotator Variance: An expert's vote should count more than a careless worker's vote. Furthermore, workers have distinct biases; one annotator might frequently confuse "cats" with "dogs," while another might systematically mislabel subtle edge cases.
  2. Instance Difficulty: Some data points are inherently ambiguous or difficult, making errors more likely, whereas others are straightforward.

To build better models, we must explicitly model why and how annotators make mistakes.

Disentangling Worker Bias from True Signal

One powerful intuition for overcoming crowd noise is to treat each annotator's response as the output of a hidden true label passed through a worker-specific "distortion filter." If a model can learn the characteristics of each worker's filter, it can effectively invert the process to recover the true label.

A foundational approach to this problem was advanced by Whitehill et al. in their work on the GLAD (Generative model of Labels, Ability, and Difficulty) framework. In Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise, the authors proposed a probabilistic model that simultaneously estimates the true label of an instance, the difficulty of that instance, and the expertise of each annotator. By modeling these latent variables together, the framework automatically downweights unreliable workers on difficult tasks while leveraging their input on easier ones.

With the advent of deep learning, researchers transitioned from separate probabilistic aggregation steps to end-to-end architectures. For instance, Filipe Rodrigues et al. introduced the Crowd Layer in Deep Learning from Crowds.

                  ┌──────────────┐
                  │ Input Image  │
                  └──────┬───────┘
                         │
                         ▼
             ┌───────────────────────┐
             │ Deep Feature Learning │
             └───────────┬───────────┘
                         │
                         ▼
             ┌───────────────────────┐
             │ Latent True Label (y) │
             └───────────┬───────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   ┌───────────┐   ┌───────────┐   ┌───────────┐
   │ Worker 1  │   │ Worker 2  │   │ Worker N  │
   │ Confusion │   │ Confusion │   │ Confusion │
   │ Matrix    │   │ Matrix    │   │ Matrix    │
   └───────────┘   └───────────┘   └───────────┘

Instead of aggregating labels beforehand, the Crowd Layer sits on top of a standard deep neural network. During training, the network outputs a prediction for the latent true label. This prediction is then multiplied by an annotator-specific confusion matrix (implemented as a specialized layer) to match the actual noisy labels provided by each individual worker. Through backpropagation, the model simultaneously learns the true task features and the unique error patterns of every annotator.

Alternatively, worker noise can be isolated using adversarial frameworks. In Adversarial Learning for Chinese NER From Crowd Annotations, Meishan Zhang et al. proposed ALCrowd for sequence labeling. The model uses an adversarial loss to separate "worker-specific" noise from "worker-invariant" features. By forcing the underlying feature extractor to learn representations that are blind to which worker provided the label, the model ensures that downstream predictions rely solely on stable linguistic patterns rather than individual annotator quirks.

Real-World Human Noise is Instance-Dependent

For a long time, theoretical research in noisy-label learning relied on synthetic noise models—such as uniformly flipping 20% of the labels to a random incorrect category. However, real human noise behaves completely differently.

In Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, Jiaheng Wei et al. collected real-world crowdsourced annotations for standard datasets, creating benchmarks like CIFAR-N. The authors demonstrated that human errors are inherently instance-dependent. A worker does not randomly mislabel a clear picture of a dog; instead, they make mistakes when an image contains ambiguous features, poor lighting, or confusing backgrounds. Because human noise is tightly coupled with the complexity of the data features themselves, algorithms designed for uniform synthetic noise often fail when deployed on real crowdsourced data.

To Aggregate or Not?

The traditional workflow for handling crowd data is a two-stage pipeline:

  1. Collect multiple annotations per instance.
  2. Aggregate them into a single consensus label (via MV or probabilistic models).
  3. Train a model on the aggregated labels.

However, recent insights suggest that this aggregation step might actually destroy valuable information. In To Aggregate or Not? Learning with Separate Noisy Labels, Jiaheng Wei et al. challenged this convention. They proved theoretically and empirically that keeping annotations separate and treating them as distinct training samples can be superior, particularly when data is sparse or noise levels are high.

When you collapse five different worker opinions into a single majority vote, you lose the underlying distribution. If three workers say "cat" and two say "dog," a hard consensus label of "cat" hides the fact that the instance is highly ambiguous. Training on the separate, raw annotations allows the loss function to naturally reflect this uncertainty, preventing the model from overconfidently learning incorrect boundaries.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 498 papers
  2. Version 1 · 2026-10-04 · 538 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_crowdsourced_label_noise_e1d2701e,
  title = {crowdsourced label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/crowdsourced-label-noise/rdr_e1d2701e47f749b3a61b2048db0c0b89},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_crowdsourced_label_noise_e1d2701e_v2,
  title = {crowdsourced label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/crowdsourced-label-noise/rdr_e1d2701e47f749b3a61b2048db0c0b89/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake