Learning to Scale Payments in Crowdsourcing with PropeRBoost

Goran Radanovic, Boi Faltings

TL;DR

This paper introduces PropeRBoost, a novel mechanism for scaling payments in crowdsourcing platforms. It effectively differentiates rewards for high-quality workers from those who provide low-quality or random data, ensuring good workers are compensated well while spammers receive minimal rewards. This is achieved through a reputation system that dynamically adjusts payment scales based on worker performance over time.

What Problem Does This Paper Address?

Crowdsourcing platforms rely on human workers to complete tasks, but a significant challenge is motivating workers to invest sufficient effort to provide accurate data. "Spammers" or low-quality workers can submit inaccurate or random responses, undermining the reliability of the collected data. Existing incentive mechanisms often struggle to effectively penalize such behavior, especially when only positive payments are allowed, as they may inadvertently reward occasional correct answers from spammers. This can lead to a small difference between the expected rewards for good work and for spamming, making it difficult to incentivize genuine effort. The paper aims to bridge this gap by designing a mechanism that creates a larger, more discernible gap between rewards for high-quality and low-quality work.

Methods

The core of PropeRBoost is a two-stage reputation protocol.

  1. Reputation-Based Scaling: A worker's reputation score dynamically influences how their payment for a task is scaled. A higher reputation leads to a scale closer to 1 (maximum payment), while a lower reputation scales the payment closer to 0 (minimal payment).
  2. Dynamic Reputation Update: The reputation score itself is updated based on the worker's performance on tasks. This performance is measured using a "performance metric" ($\pi_w$) that quantifies how accurate a worker's response is compared to an estimated ground truth. This metric is then used to calculate a "score" ($score_w$), which in turn updates the reputation.

The mechanism uses an estimator ($\hat{F}$) to estimate the correct answer for tasks, which is assumed to be better than random guessing. The performance metric $\pi_w$ is designed to be linear in the worker's proficiency ($p$), allowing for the definition of a "score function" that is also linear in proficiency. This score function is then used in an exponential update rule for the reputation: $p_w \leftarrow p_w \cdot (1 + \eta \cdot score_w)$, where $\eta$ is a learning rate. This update rule ensures that positive scores boost reputation and future payments, while negative scores decrease them, effectively penalizing poor performance over time.

Datasets

The paper uses synthetic datasets for simulations.

  • Multi-task Crowdsourcing: A synthetic dataset where each task has a binary answer. 50 workers with proficiencies drawn uniformly from [0.5, 1] are simulated. Each worker is assigned 12 tasks from a bundle of 100, repeated for 500 time steps.
  • Crowd-sensing: A real-world dataset of UFP (ultra fine particles) measurements over Zurich. This dataset is used to simulate a crowd-sensing scenario where a worker's measurement is compared to a trusted sensor. The data is processed to create binary signals for performance evaluation.

Key Insights and Ideas

  • Reputation as a Dynamic Payment Scaler: The core idea is to use a worker's accumulated reputation, built over time, to dynamically adjust their payment scale. This allows for a more nuanced and adaptive incentive system than static payment rules.
  • Separating Good vs. Bad Performance: PropeRBoost explicitly aims to maximize the difference between the expected payment for good workers and spammers. This is crucial because low-quality work often requires less effort, making it potentially more profitable if not properly penalized.
  • Penalizing Bad Behavior Through Future Payments: By linking current performance to future reputation and thus future payment scales, PropeRBoost can effectively penalize spammers over time. A single bad task can slightly reduce reputation, leading to a slightly lower payment for the next task, creating a compounding effect.
  • Bounded Negative Gain: Even with low-quality workers, the mechanism's overall expected gain is bounded from below, meaning it's unlikely to incur significant losses compared to not having a reward system at all.
  • Incentivizing Honest Reporting: The mechanism is designed to ensure that honest workers with sufficiently high proficiency receive payments close to the maximum possible, demonstrating its effectiveness in rewarding good behavior.

Key Results

  • PropeRBoost vs. Baseline: Simulations show that PropeRBoost provides a much clearer separation between rewards for honest and random strategies compared to a traditional peer-consistency mechanism (Dasgupta and Ghosh, 2013).
  • Scale Convergence: For honest workers, the payment scale converges towards 1. For random workers, the scale converges towards 0.
  • Strategy-Dependent Scales: Different worker strategies (e.g., "switch" between honest and random, "keepRep" to maintain reputation) result in distinct scale evolutions, illustrating the mechanism's responsiveness.
  • Crowd-sensing Effectiveness: In a crowd-sensing scenario, PropeRBoost also demonstrates significantly more separated payments between honest and random reporting compared to a quadratic scoring rule.
  • Bounded Low-Quality Payments: Theoretical results show that workers with proficiency below a certain threshold ($p < p_i$) receive expected payments close to 0 in the long run.

Limitations and Future Work

  • Experimental Comparison: The authors acknowledge that a significant future step is to experimentally compare PropeRBoost with existing incentive mechanisms in real-world settings. This is challenging because PropeRBoost relies on repeated interactions, unlike many single-shot mechanisms.
  • Parameter Tuning: The performance of PropeRBoost depends on parameters like $\eta$ (learning rate) and thresholds $p_i, p_u$. While theoretical guidance is provided, optimal tuning might require empirical investigation.
  • Assumptions on Estimator: The mechanism relies on an estimator $\hat{F}$ that is better than random. The quality of this estimator directly impacts the effectiveness of the performance metric and reputation updates.
  • Manipulation: While the paper focuses on honest workers and theoretical guarantees, the potential for sophisticated manipulation strategies by workers is not fully explored.

Summary updated

BibTeX
@inproceedings{radanovic2016learning,
  author = {Goran Radanovic and Boi Faltings},
  title = {{Learning to Scale Payments in Crowdsourcing with PropeRBoost}},
  booktitle = {HCOMP},
  year = {2016},
  doi = {10.1609/hcomp.v4i1.13279},
}

Loading related content...

Figure