Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing

Alexandry Augustin, Matteo Venanzi, Alex Rogers, Nicholas R. JenningsIJCAI 2017 · DOI

TL;DR

This paper introduces MBCC, a novel Bayesian framework for aggregating categorical judgments from crowdsourcing platforms, specifically designed to handle "spammers" who intentionally provide inaccurate data. MBCC achieves comparable aggregation accuracy to existing methods even when up to 60% of workers are spammers, while also significantly improving the detection of these malicious contributors.

What Problem Does This Paper Address?

Crowdsourcing platforms like Amazon Mechanical Turk have democratized data collection, enabling rapid and low-cost annotation for tasks ranging from sentiment analysis to digital humanitarianism. A common challenge is aggregating "judgments of proportions," where workers estimate the percentage of different categories within an item (e.g., the proportion of topics in a document or colors in an image). This aggregation is crucial for obtaining a reliable consensus view. However, a significant problem arises from "spammers" – workers who intentionally provide random or misleading judgments, often for financial gain. These spammers not only inflate costs but also severely degrade the accuracy of aggregated results. Existing methods, while effective in clean datasets, struggle to robustly handle the presence of spammers.

Methods

The paper proposes Multi-Category Independent Bayesian Classifier Combination (MBCC), a Bayesian model that extends the existing IBCC framework. The core innovation lies in its ability to:

  • Model Proportions Directly: Instead of assuming each document belongs to a single category, MBCC models the proportion of categories within a document using a categorical distribution.
  • Handle Multiple Categories per Item: It extends aggregation to scenarios where items can have multiple categories simultaneously.
  • Incorporate Spammer Detection: MBCC learns individual worker accuracies through confusion matrices. By modeling these matrices as random variables with Dirichlet priors, it can infer which workers are likely spammers.
  • Generative Process: The model defines a generative process where document category proportions are sampled, and then worker judgments are sampled based on their learned confusion matrices. This process is summarized in Figure 1 and Algorithm 1.
  • Inference: The paper utilizes variational message passing for inference, citing its speed and accuracy advantages over sampling methods.

Datasets

The authors evaluated MBCC on three datasets:

  • SemEval: Judgments of sentiments within news headlines.
  • IAPR-TC12: Estimating the proportion of regions in urban and rural scene images.
  • Colours: Estimating the proportion of colors in national flags.

These datasets were augmented with synthetic spammers to simulate real-world scenarios with varying levels of malicious activity.

Key Insights and Ideas

  • Spammers are a Major Hurdle: The presence of even a moderate percentage of spammers can drastically reduce the accuracy of standard aggregation methods.
  • Confusion Matrices are Key: Modeling individual worker accuracy via confusion matrices is essential for robust aggregation and spammer detection.
  • Bayesian Approach for Flexibility: The Bayesian framework allows for flexible modeling of worker abilities and document properties, particularly through the use of prior distributions.
  • MBCC's Robustness: MBCC demonstrates remarkable resilience to spammers, maintaining accuracy where other methods falter.
  • Improved Spam Detection: Beyond aggregation, MBCC offers a significant advantage in identifying and classifying spammers.

Key Results

  • Aggregation Accuracy: MBCC achieved comparable aggregation accuracy to state-of-the-art methods even when 60% of workers were spammers, a feat unmatched by other approaches under such conditions.
  • Spammer Detection Improvement: MBCC showed a 5-fold improvement in detecting misclassified spammers compared to the IBCC method, as evidenced by ROC curves with an AUC of 0.99.
  • Performance Trade-offs: While MBCC excels with spammers, simpler methods like LinOp might offer slightly higher accuracy when the dataset is clean (few or no spammers), but MBCC quickly overtakes them as spammer presence increases.
  • Dataset Specifics: Performance varied across datasets; for instance, IBCC performed well on IAPR-TC12 due to its single-dominant category nature, while MBCC showed consistent improvement on the Colours dataset.
  • Computational Cost: MBCC and IBCC are computationally more intensive than LinOp and Median, with running times ranging from seconds to minutes depending on the number of samples used.

Limitations and Future Work

  • Computational Cost: While effective, MBCC is computationally more demanding than simpler methods, especially with a large number of samples.
  • Prior Sensitivity: The performance of MBCC can be influenced by the choice of prior distributions for the confusion matrices. The paper notes that adjusting these priors can tune the model's sensitivity to spammers.
  • Future Work: The authors suggest exploring other approximation methods for inference and potentially extending the model to handle more complex judgment types or worker behaviors. The paper also hints at the possibility of using the spammer classification to actively remove or down-weight malicious workers in future tasks.

Summary updated

BibTeX
@inproceedings{augustin2017bayesian,
  author = {Alexandry Augustin and Matteo Venanzi and Alex Rogers and Nicholas R. Jennings},
  title = {{Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/195},
}
Figure