Parallel and Streaming Truth Discovery in Large-Scale Quantitative Crowdsourcing

Wentao Robin Ouyang, Lance M. Kaplan, Alice Toniolo, Mani B. Srivastava, Timothy J. NormanIEEE TPDS 2016 · DOI · OpenReview

Abstract

To enable reliable crowdsourcing applications, it is of great importance to develop algorithms that can automatically discover the truths from possibly noisy and conflicting claims provided by various information sources. In order to handle crowdsourcing applications involving big or streaming data, a desirable truth discovery algorithm should not only be <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effective</i> , but also be <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable</i> . However, with respect to quantitative crowdsourcing applications such as object counting and percentage annotation, existing truth discovery algorithms are not simultaneously effective and scalable. They either address truth discovery in categorical crowdsourcing or perform batch processing that does not scale. In this paper, we propose new parallel and streaming truth discovery algorithms for quantitative crowdsourcing applications. Through extensive experiments on real-world and synthetic datasets, we demonstrate that 1) both of them are quite effective, 2) the parallel algorithm can efficiently perform truth discovery on large datasets, and 3) the streaming algorithm processes data incrementally, and it can efficiently perform truth discovery both on large datasets and in data streams.
BibTeX
@article{ouyang2016parallel,
  author = {Wentao Robin Ouyang and Lance M. Kaplan and Alice Toniolo and Mani B. Srivastava and Timothy J. Norman},
  title = {{Parallel and Streaming Truth Discovery in Large-Scale Quantitative Crowdsourcing}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2016},
  doi = {10.1109/tpds.2016.2515092},
}

Loading related content...

Figure