Identifying Careless Workers in Crowdsourcing Platforms: A Game Theory Approach
Lists
Only you see this
Abstract
In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.
BibTeX
@inproceedings{moshfeghi2016identifying,
author = {Yashar Moshfeghi and Alvaro Francisco Huertas-Rosero and Joemon M. Jose},
title = {{Identifying Careless Workers in Crowdsourcing Platforms: A Game Theory Approach}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2016},
doi = {10.1145/2911451.2914756},
}Loading related content...