Data Poisoning Attacks and Defenses to Crowdsourcing Systems

Minghong Fang, Minghao Sun, Qi Li, Neil Zhenqiang Gong, Jin Tian, Jia Liu

TL;DR

This paper explores the vulnerability of crowdsourcing systems to data poisoning attacks, where malicious users inject crafted data to corrupt aggregated results. The authors propose novel attack strategies that significantly increase estimation errors and introduce two defense mechanisms to mitigate these attacks, demonstrating their effectiveness in reducing error.

What Problem Does This Paper Address?

Big data analytics often relies on crowdsourcing to efficiently collect large volumes of labeled data. Crowdsourcing platforms typically aggregate data from multiple users to improve accuracy, assuming users are generally honest but may make mistakes. However, this paper addresses a critical security gap: the vulnerability of these systems to data poisoning attacks. In such attacks, malicious users (or attackers controlling them) deliberately provide false or misleading data to manipulate the final aggregated outcome. While previous work has explored data poisoning in other machine learning contexts and for categorical crowdsourced data, this paper specifically investigates and tackles these attacks in the context of continuous-valued crowdsourcing data, which is prevalent in many real-world applications (e.g., sensor readings, sentiment scores). The core problem is that existing truth discovery methods, designed to handle noisy data, can be severely compromised by strategically crafted malicious inputs, leading to inaccurate and potentially harmful aggregated results.

Methods

The paper introduces two main categories of methods: attacks and defenses.

Attacks: * Optimization-based Formulation: The core attack strategy is formulated as a bi-level optimization problem. The attacker aims to maximize the estimation error of aggregated values for specific "targeted" items. This involves finding optimal values for malicious workers' submissions that, when processed by the crowdsourcing system's truth discovery algorithm, lead to the largest deviation from the true aggregated value. * Full-Knowledge Attack: In this scenario, the attacker has complete knowledge of the aggregation algorithm (e.g., CRH, GTM) and all data provided by normal workers. The attack iteratively updates malicious workers' values to maximize the objective function, while ensuring these values remain within a plausible range to avoid detection. * Partial-Knowledge Attack: This more realistic scenario assumes the attacker only knows the aggregation algorithm and a subset of normal workers' data for the targeted items. To overcome the lack of complete information, the attacker employs a bootstrapping technique to estimate the "before-attack" aggregated values more robustly, which then aids in crafting more effective malicious inputs.

Defenses: * Median-of-Weighted-Average (MWA): This defense partitions workers into groups, computes weighted averages within each group, and then takes the median of these group averages as the final aggregated value. This approach leverages robust aggregation to mitigate the impact of malicious data. * Maximize Influence of Estimation (MIE): This defense aims to identify and remove potentially malicious workers. It quantifies the "influence" of each worker on the overall estimation and removes a predetermined number of the most influential workers, assuming they are malicious. This method relies on a greedy approximation algorithm due to the NP-hard nature of influence maximization.

Datasets

The authors evaluated their methods using:

  • One Synthetic Dataset: This dataset was generated with controlled parameters (ground truth, worker reliability) to demonstrate the effectiveness of attacks and defenses in a controlled environment. It contained 50,000 values across 4,000 items from 500 workers.
  • Two Real-World Datasets:
    • Emotion [37]: This dataset involves assigning sentiment scores (ranging from -100 to 100) to text documents. It has 700 items and 38 workers.
  • Weather [42]: This dataset contains temperature forecast information for US cities. It has 7,568 items and 152 workers.

Key Insights and Ideas

  • Crowdsourcing is Vulnerable: The paper convincingly demonstrates that even sophisticated truth discovery methods, designed to handle noisy data, are susceptible to targeted data poisoning attacks, especially with continuous data.
  • Attacker's Goal: Maximize Error: The core insight is that attackers don't just aim to provide wrong answers, but to maximize the error of the aggregated result, making the system's output as unreliable as possible.
  • Bi-level Optimization for Attacks: Formulating the attack as a bi-level optimization problem elegantly captures the attacker's dual goal: manipulating data to fool the aggregation algorithm while ensuring the manipulated data itself appears plausible.
  • Bootstrapping for Partial Knowledge: The use of bootstrapping in the partial-knowledge attack is a clever way to estimate the system's state without full information, significantly enhancing attack effectiveness in more realistic scenarios.
  • Robust Aggregation vs. Worker Removal: The two proposed defenses represent different philosophies: MWA uses robust aggregation to dilute the impact of bad data, while MIE actively tries to identify and remove suspected malicious workers.
  • Malicious Workers Can Blend In: The experiments show that well-crafted attacks can make malicious workers' weights/variances indistinguishable from normal workers, highlighting the difficulty of detection.

Key Results

  • Attack Effectiveness: The proposed data poisoning attacks significantly increase the average estimation error. For instance, on the Emotion dataset, the attack increased the error to 93.69% with only 10% malicious workers under the CRH model.
  • Attack Performance: The full-knowledge and partial-knowledge attacks consistently outperformed baseline "random" and "maximum" attacks, demonstrating the value of optimization-based strategies tailored to the crowdsourcing system.
  • Defense Effectiveness: Both MWA and MIE defenses substantially reduced the estimation errors caused by the attacks. MIE generally showed better performance than MWA.
  • MIE Superiority: MIE was found to be more effective than MWA, though it requires stronger assumptions about the attacker's presence and number.
  • Weight/Variance Blending: Figures 4-7 show that malicious workers generated by the full-knowledge attack had weights/variances that were hard to distinguish from normal workers, indicating successful evasion of typical reliability metrics.

Limitations and Future Work

  • Computational Complexity: The bi-level optimization problem for attacks is NP-hard, requiring iterative approximation methods. Similarly, the MIE defense's influence maximization component is also NP-hard, necessitating greedy approximations.
  • Assumptions for MIE: The MIE defense assumes the server knows the system is attacked and the number of malicious workers, which might not always hold.
  • Defense Vulnerability: The paper notes that even with defenses, if the percentage of malicious workers is very high, the system's robustness can be compromised (e.g., MWA's performance degrades with more malicious workers).
  • Future Work: The authors suggest exploring more sophisticated attack strategies and defenses, potentially incorporating adaptive mechanisms that can detect and respond to evolving attack patterns. Further research into the trade-offs between different defense strategies and their computational costs would also be valuable.

Summary updated

Abstract

A key challenge of big data analytics is how to collect a large volume of (labeled) data. Crowdsourcing aims to address this challenge via aggregating and estimating high-quality data (e.g., sentiment label for text) from pervasive clients/users. Existing studies on crowdsourcing focus on designing new methods to improve the aggregated data quality from unreliable/noisy clients. However, the security aspects of such crowdsourcing systems remain under-explored to date. We aim to bridge this gap in this work. Specifically, we show that crowdsourcing is vulnerable to data poisoning attacks, in which malicious clients provide carefully crafted data to corrupt the aggregated data. We formulate our proposed data poisoning attacks as an optimization problem that maximizes the error of the aggregated data. Our evaluation results on one synthetic and two real-world benchmark datasets demonstrate that the proposed attacks can substantially increase the estimation errors of the aggregated data. We also propose two defenses to reduce the impact of malicious clients. Our empirical results show that the proposed defenses can substantially reduce the estimation errors of the data poisoning attacks.
BibTeX
@inproceedings{fang2021data,
  author = {Minghong Fang and Minghao Sun and Qi Li and Neil Zhenqiang Gong and Jin Tian and Jia Liu},
  title = {{Data Poisoning Attacks and Defenses to Crowdsourcing Systems}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3450066},
}
Figure