Label Selection Approach to Learning from Crowds

Kosuke Yoshimura, Hisashi Kashima

Introduction

Deep learning models are notoriously data-hungry, often requiring massive datasets that are typically labeled via crowdsourcing platforms like Amazon Mechanical Turk. However, the quality of these labels is rarely uniform; crowd workers vary significantly in their expertise, attention spans, and inherent biases. This variability introduces "noise" into the training data, which can degrade model performance if not properly managed.

The standard paradigm for "Learning from Crowds" involves training models directly on this noisy data while attempting to account for worker reliability. Traditional state-of-the-art methods, such as the Crowd Layer, rely on generative noise models. These models attempt to explicitly estimate the probability of a worker mislabeling a specific class—for example, calculating the likelihood of a worker confusing a "Cat" for a "Dog." While effective for simple classification, these hand-designed noise models become prohibitively complex for structured output tasks like Named Entity Recognition (NER) or translation, where the error space is not a simple class-to-class swap.

The Label Selection Layer (LSL)

To address these limitations, this research introduces the Label Selection Layer (LSL). LSL shifts the focus from modeling how a worker is wrong to determining whether a worker's label should be trusted for a specific instance. This approach is model-agnostic and can be integrated into various neural network architectures.

Inspired by SelectiveNet—a framework designed for selective prediction where a model can reject low-confidence inputs—LSL adds a "selector" component to the network. This selector outputs a weight between 0 and 1 for every worker-label pair. The training loss is then weighted by this selection score. To prevent the model from simply ignoring all labels to minimize loss, the authors utilize a hyperparameter $c$ (target coverage), which forces the model to "adopt" a minimum percentage of the available annotations.

Selector Variants

The authors proposed four distinct variations of the selector function $g$ to accommodate different data types and worker behaviors:

  1. Simple: Assigns a single learnable weight per worker, assuming their reliability is constant across all tasks.
  2. Class-wise: Assigns weights based on both the worker and the specific label provided, accounting for workers who may be experts in one category but unreliable in another.
  3. Target-wise: Specifically designed for regression tasks, utilizing a worker's weight and bias relative to continuous target values.
  4. Feature-based: The most sophisticated variant, where selection depends on the input features of the data itself. This allows the model to learn that certain workers are only reliable for specific types of examples (e.g., nighttime vs. daytime images).

Experimental Evaluation

The LSL was evaluated across three diverse problem domains to test its versatility:

  • Image Classification (LabelMe): Using a dataset of 8 classes with labels from 59 workers, the Feature-based LSL achieved an accuracy of 0.839, outperforming all Crowd Layer variants, which peaked at approximately 0.824.
  • Named Entity Recognition (CoNLL-2003): In this complex sequence labeling task involving 49 workers, LSL variants demonstrated a significant advantage in precision. LSL reached up to 64.03% precision, compared to the 54.95% achieved by the best Crowd Layer variant. This suggests that LSL is particularly adept at "cherry-picking" high-certainty labels in structured tasks.
  • Rating Regression (Movie Reviews): In this domain (scores 0-10, 135 workers), LSL struggled relative to the Crowd Layer. The authors noted that LSL's binary-style selection (use or don't use) cannot account for workers who are consistently "offset" (e.g., a worker who always rates movies 2 points lower than the average). The Crowd Layer's bias parameters handle these systematic shifts more naturally.

Key Insights and Future Directions

The primary advantage of LSL is its elimination of the need for complex generative noise models. By treating reliability as a selection problem, LSL can be plugged into almost any architecture without prior knowledge of worker error distributions. The "Feature-based" results further highlight that worker reliability is often context-dependent, a nuance that LSL captures effectively.

However, the method's current inability to "correct" labels—only select them—limits its utility in regression tasks where systematic bias is prevalent. Additionally, performance remains sensitive to the target coverage hyperparameter $c$. Future research will focus on automating the tuning of this adoption rate and extending LSL to even more complex structured data formats.

Summary updated

BibTeX
@article{yoshimura2024label,
  author = {Kosuke Yoshimura and Hisashi Kashima},
  title = {{Label Selection Approach to Learning from Crowds}},
  journal = {Transactions},
  year = {2024},
  doi = {10.1527/tjsai.39-5\_f-o23},
}
Figure