Weak Labeling for Crowd Learning

Iker Beñaran-Muñoz, Jerónimo Hernández-González, Aritz PérezarXiv 2018 · arXiv

Introduction

Supervised machine learning relies heavily on high-quality ground truth labels, which are often obtained through crowdsourcing platforms. However, non-expert workers frequently encounter tasks where they are uncertain, leading to noisy data when forced to choose a single label—a process known as full labeling. This paper introduces candidate labeling, an approach that allows workers to provide a set of likely labels, capturing partial knowledge and reducing noise.

Methodology

The authors propose a framework for candidate labeling and a corresponding aggregation method called candidate voting. In this scheme, each annotator provides a set $S$ of candidate labels. These sets are aggregated using a weighted voting function: $$\nu(L) = \arg \max_c \sum_{j=1}^l \frac{1}{|S_j|} \mathbf{1}(c \in S_j)$$ This ensures that a worker's vote is distributed across their chosen candidates, effectively weighting their certainty.

To evaluate this approach, the researchers developed a simulation environment that models instance difficulty and annotator "hesitation"—the degree to which an annotator's internal probability distribution aligns with the actual task difficulty.

Overview of the experimental framework used to simulate and evaluate candidate labeling.

Key Results

The study compared candidate labeling against traditional full labeling using both synthetic and real-world datasets (including UCI datasets like MNIST and arrhythmia).

Performance and Efficiency

The primary finding is that candidate labeling ($s > 1$) consistently outperforms full labeling ($s = 1$). As shown in the error curves below, increasing the number of allowed candidates significantly reduces the error rate, especially in complex domains with many classes.

Error curves demonstrating that candidate labeling (s > 1) achieves lower error rates than full labeling (s = 1) across various settings.

Key observations include:

  • Force Multiplier: In difficult tasks, 5 candidate-labeling workers can achieve the same accuracy as 15-20 workers using traditional methods.
  • Hesitation Advantage: Unlike full labeling, which suffers when workers are hesitant, candidate labeling thrives by capturing the richer information contained in a worker's uncertainty.
  • Variance Reduction: Analysis shows that candidate labeling primarily improves performance by reducing the variance of aggregated labels, making the system less sensitive to random guesses in forced-choice scenarios.

Conclusion

Candidate labeling offers a more efficient and accurate alternative to traditional crowdsourcing methods. By allowing workers to express uncertainty, researchers can extract more information per annotator, reaching performance levels close to theoretical bounds. Future work will focus on real-world deployment and accounting for worker reliability in more complex classification tasks.

Summary updated

Abstract

Crowdsourcing has become very popular among the machine learning community as a way to obtain labels that allow a ground truth to be estimated for a given dataset. In most of the approaches that use crowdsourced labels, annotators are asked to provide, for each presented instance, a single class label. Such a request could be inefficient, that is, considering that the labelers may not be experts, that way to proceed could fail to take real advantage of the knowledge of the labelers. In this paper, the use of candidate labeling for crowd learning is proposed, where the annotators may provide more than a single label per instance to try not to miss the real label. The main hypothesis is that, by allowing candidate labeling, knowledge can be extracted from the labelers more efficiently by than in the standard crowd learning scenario. Empirical evidence which supports that hypothesis is presented.
BibTeX
@misc{benaranmunoz2018weak,
  author = {Iker Beñaran-Muñoz and Jerónimo Hernández-González and Aritz Pérez},
  title = {{Weak Labeling for Crowd Learning}},
  year = {2018},
  eprint = {1804.10023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.10023},
}

Loading related content...

Figure