Bonus or Not? Learn to Reward in Crowdsourcing

Ming Yin, Yiling Chen

Introduction

In crowdsourcing marketplaces like Amazon Mechanical Turk (MTurk), requesters face a persistent challenge: balancing the need for high-quality output with the constraint of limited budgets. While financial bonuses are a powerful tool to motivate workers, their deployment is often inefficient. Most current practices rely on static policies—offering bonuses to everyone, no one, or at random intervals—ignoring the dynamic and heterogeneous nature of worker behavior. This paper proposes a move toward personalized, dynamic incentive design that reacts to real-time performance to maximize utility.

Algorithmic Framework

The authors introduce a two-step framework to optimize bonus allocation:

  1. Worker Modeling with IOHMM: The researchers use an Input-Output Hidden Markov Model (IOHMM) to capture the nuances of worker behavior. In this model, the Input is the presence or absence of a bonus, the Output is the observed work quality, and the Hidden State represents the worker's internal motivation or their "reference" wage expectation. This allows the system to infer whether a worker is naturally diligent or only exerts effort when incentivized.
  2. Decision Making via POMDP: With the worker model established, the task of timing bonuses is framed as a Partially Observable Markov Decision Process (POMDP). To manage the computational complexity of solving POMDPs, the authors evaluate three heuristic strategies:
  3. $n$-step look-ahead: Short-term planning for the next few tasks.
  4. MLS-MDP (Most Likely Sequence): Acting based on the most probable inferred hidden state.
  5. Q-MDP: Weighting optimal actions across all possible states by their current probability.

Experimental Results

Real-World Performance on MTurk

The framework was tested on a word puzzle task involving 300 new workers. The results demonstrated that dynamic policies significantly outperform traditional baselines. Specifically, the MLS-MDP heuristic emerged as the most effective, increasing requester utility by 27.22% compared to the "No Bonus" baseline.

Figure 1: Comparison of utility, high-quality (HQ) answers, and costs across different incentive schemes on MTurk.

Beyond raw utility, the algorithmic approach proved highly cost-efficient. The 2-step look-ahead (2-LA) policy achieved work quality comparable to a random 50% bonus scheme while reducing bonus expenditures by 77.9%. This efficiency stems from the model's ability to identify "diligent" workers who provide high quality without extra pay, while strategically using "rescue" bonuses to re-engage workers whose performance begins to slip.

Robustness and Simulation

To ensure the findings were not limited to a single population, the authors conducted simulations across different worker distributions (Uniform and Beta distributions). These simulations confirmed that the dynamic policies consistently maintain higher utility than static baselines, even as the cost of bonuses ($c$) increases.

Figure 3: Simulation results showing utility across varying bonus costs (c) for two different worker populations.

Discussion and Future Work

While the results are promising, the authors highlight several areas for future exploration. A primary concern is long-term fairness: if an algorithm only rewards workers who slack off without bonuses, it may inadvertently penalize and alienate naturally high-performing workers. Additionally, the current model assumes effort-responsive tasks and binary quality metrics. Future iterations could expand this to expert-level tasks where quality depends more on skill than effort, and incorporate continuous quality scoring for more granular optimization.

Summary updated

Abstract

Recent work has shown that the quality of work produced in a crowdsourcing working session can be influenced by the presence of performance-contingent financial incentives, such as bonuses for exceptional performance, in the session. We take an algorithmic approach to decide when to offer bonuses in a working session to improve the overall utility that a requester derives from the session. Specifically, we propose and train an input-output hidden Markov model to learn the impact of bonuses on work quality and then use this model to dynamically decide whether to offer a bonus on each task in a working session to maximize a requester's utility. Experiments on Amazon Mechanical Turk show that our approach leads to higher utility for the requester than fixed and random bonus schemes do. Simulations on synthesized data sets further demonstrate the robustness of our approach against different worker population and worker behavior in improving requester utility.
BibTeX
@inproceedings{yin2015bonus,
  author = {Ming Yin and Yiling Chen},
  title = {{Bonus or Not? Learn to Reward in Crowdsourcing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}

Loading related content...

Figure