Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

Guangyang Han, Sufang Li, Runmin Wang, Chunming Wu

Abstract

Crowdsourcing is an online outsourcing mode which can solve the current machine learning algorithm's urge need for massive labeled data. Requester posts tasks on crowdsourcing platforms, which employ online workers over the Internet to complete tasks, then aggregate and return results to requester. How to model the interaction between different types of workers and tasks is a hot spot. In this paper, we try to model workers as four types based on their ability: expert, normal worker, sloppy worker and spammer, and divide tasks into hard, medium and easy task according to their difficulty. We believe that even experts struggle with difficult tasks while sloppy workers can get easy tasks right, and spammers always give out wrong answers deliberately. So, good examination tasks should have moderate degree of difficulty and discriminability to score workers more objectively. Thus, we first score workers' ability mainly on the medium difficult tasks, then reducing the weight of answers from sloppy workers and modifying the answers from spammers when inferring the tasks' ground truth. A probability graph model is adopted to simulate the task execution process, and an iterative method is adopted to calculate and update the ground truth, the ability of workers and the difficulty of the task successively. We verify the rightness and effectiveness of our algorithm both in simulated and real crowdsourcing scenes.

Introduction

Modern machine learning relies heavily on large-scale labeled datasets, often sourced through crowdsourcing platforms like Amazon Mechanical Turk. However, the reliability of crowd workers varies significantly, ranging from experts to "sloppy" workers and malicious spammers. Traditional truth inference methods, such as Majority Vote, often fail to account for the inherent difficulty of the tasks themselves. This paper proposes a novel framework that treats crowdsourcing as an examination process, borrowing principles from educational psychometrics to jointly estimate worker ability and task difficulty.

The Worker-Task Interaction Model (WTIM)

The core of the proposed approach is the Worker-Task Interaction Model (WTIM), which uses a Probability Graph Model (PGM) to represent the relationships between latent variables and observed data.

The Probability Graph Model (PGM) for the Worker-Task Interaction Model (WTIM).

In this model, a worker's confidence $p^i_j$ in providing a correct answer for task $i$ is a function of their latent ability $e_j$ and the task's difficulty $d^i$. The model categorizes workers into four types: experts, normal workers, sloppy workers, and spammers. Similarly, tasks are categorized by difficulty: easy, medium, and hard.

A critical insight of WTIM is the "Goldilocks" principle of discriminability. Tasks that are too easy or too hard provide little information about a worker's true skill. As shown in the interaction curves, tasks with moderate difficulty ($d \approx 0.4$) are the most effective at discriminating between different levels of worker ability.

Interaction between worker ability (e) and task difficulty (d) on the probability of a correct answer (p).

To infer the ground truth, the authors employ an Expectation-Maximization (EM) style iterative algorithm. It begins with a Majority Vote baseline and iteratively refines estimates for task difficulty and worker ability until convergence. During this process, the algorithm strategically weights performance on medium-difficulty tasks more heavily to better identify expert workers.

Experimental Evaluation

The WTIM was validated using three types of datasets:

  1. Simulated Datasets: Controlled environments with 100 tasks and 10 workers, testing both binary and 4-choice scenarios.
  2. Real-World Crowdsourcing Data:
  3. BM (Barzan Mozafari) Dataset: 1,000 binary classification tasks with 5,000 annotations.
  4. AC (Adult Content) Dataset: 333 tasks involving 4-class safety labeling with known ground truth.
  5. Real-World Quiz: A 20-question quiz administered to 10 computer science students to validate the "examination" metaphor in a literal setting.

Results and Discussion

Experimental results demonstrate that WTIM significantly outperforms baseline methods. In simulated 4-choice tasks, WTIM achieved 69% accuracy, compared to 52% for Majority Vote and 66% for the state-of-the-art GLAD model. On the real-world Adult Content dataset, WTIM reached 77.17% accuracy, slightly surpassing GLAD (76.87%).

The study also highlighted the "cost of precision." WTIM is computationally more intensive than simpler methods; for instance, it took over 4,500 seconds to converge on the AC dataset, whereas Majority Vote took less than a second. Furthermore, the model's effectiveness is tied to the redundancy of annotations, performing best when each task has at least five annotations ($r \ge 5$).

Conclusion

By treating crowdsourcing as an examination, WTIM provides a robust framework for truth inference that accounts for both worker ability and task difficulty. While the computational overhead is higher, the gains in accuracy—especially in complex multi-choice tasks—make it a valuable tool for high-quality data labeling. Future research will focus on optimizing the model's efficiency and incorporating human-centric factors like task design and incentive structures directly into the probabilistic framework.

Summary updated

Abstract

Crowdsourcing is an online outsourcing mode which can solve the current machine learning algorithm's urge need for massive labeled data. Requester posts tasks on crowdsourcing platforms, which employ online workers over the Internet to complete tasks, then aggregate and return results to requester. How to model the interaction between different types of workers and tasks is a hot spot. In this paper, we try to model workers as four types based on their ability: expert, normal worker, sloppy worker and spammer, and divide tasks into hard, medium and easy task according to their difficulty. We believe that even experts struggle with difficult tasks while sloppy workers can get easy tasks right, and spammers always give out wrong answers deliberately. So, good examination tasks should have moderate degree of difficulty and discriminability to score workers more objectively. Thus, we first score workers' ability mainly on the medium difficult tasks, then reducing the weight of answers from sloppy workers and modifying the answers from spammers when inferring the tasks' ground truth. A probability graph model is adopted to simulate the task execution process, and an iterative method is adopted to calculate and update the ground truth, the ability of workers and the difficulty of the task successively. We verify the rightness and effectiveness of our algorithm both in simulated and real crowdsourcing scenes.
BibTeX
@misc{han2022treating,
  author = {Guangyang Han and Sufang Li and Runmin Wang and Chunming Wu},
  title = {{Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?}},
  year = {2022},
  eprint = {2204.13065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.13065},
}
Figure