HiveMind: Tuning Crowd Response with a Single Value

Preetjot Singh, Walter S. Lasecki, Paulo Barelli, Jeffrey P. BighamHCOMP (Works in Progress / Demos) 2013 · OpenReview

Abstract

One common problem plaguing crowdsourcing tasks is tuning the set of worker responses: Depending on task requirements, requesters may want a large set of rich and varied worker responses (typically in subjective evaluation tasks) or a more convergent response-set (typically for more objective tasks such as fact-checking). This problem is especially salient in tasks that combine workers’ responses to present a single output: Divergence in these settings could either add richness and complexity to the unified answer, or noise. In this paper we present HiveMind, a system of methods that allow requesters to tune different levels of convergence in worker participation for different tasks simply by adjusting the value of one variable.

Introduction

Crowdsourcing tasks are not one-size-fits-all. While image labeling or fact-checking requires the crowd to settle on a single, correct answer (convergence), creative brainstorming or subjective evaluation benefits from a rich, varied set of responses (divergence). Currently, tuning this balance is difficult because requesters often struggle to filter out low-effort "noise" while encouraging high-quality "richness."

HiveMind addresses this by shifting the burden of evaluation to the workers themselves. By using a game-theoretic payment model, the system incentivizes workers to self-evaluate their own confidence and choose the action—Agree, Propose, or Abstain—that best aligns with the requester's goals. This is controlled by a single parameter, $\alpha$, which acts as a "control knob" for the entire system.

Methodology: The Stepwise Structure and Pivot Algorithm

HiveMind introduces a two-part framework to manage crowd input:

  1. Stepwise Structure: As workers enter a task, they see a list of current responses. They have three choices: Agree (vote for an existing response), Propose (suggest a new response), or Abstain (do nothing). This allows the input to evolve over time as a "cascade" of refined ideas.
Stepwise evolution of crowd responses across worker groups
  1. The Pivot Algorithm: This incentive structure requires workers to participate in a Vickrey auction to "bid" for the task, ensuring they have a financial stake ($b$) in the outcome. If a worker's choice becomes the majority or is deemed correct, they receive a reward; if they are wrong, they lose their stake. The variable $\alpha$ scales the reward for proposing a new idea. By lowering $\alpha$, the requester makes the potential payout for a successful "Propose" action much higher, encouraging workers to suggest new ideas even if they aren't 100% certain.

Analysis of Worker Behavior

A core philosophy of HiveMind is that the worker is the best judge of their own capability. By making the cost of being wrong high, the system forces workers to only act when their internal confidence exceeds a mathematical threshold.

Temporal Optimization

Because workers have negative expected payoffs at the very start of a task (before enough information is available), they will naturally wait until the "optimal" time ($t_1$) to contribute. This improves the quality of the evolution of the answers by ensuring early participants are highly confident or that the crowd waits for sufficient context.

Temporal optimization of worker participation based on expected payoff

The $\alpha$ Control Knob

The paper provides mathematical proofs demonstrating that moving $\alpha$ directly changes the confidence threshold required for a worker to suggest something new. For example, a worker will only choose to propose a new idea over agreeing with an existing one if their confidence in the new idea ($\eta$) is at least $2\mu\alpha$ (where $\mu$ is confidence in the majority).

  • For Convergence: Set $\alpha$ high to make it risky to propose new things.
  • For Divergence: Set $\alpha$ low to reward successful novelty.

Conclusion and Future Work

HiveMind offers a theoretical framework to eliminate "freeloaders"—who are financially disincentivized from making arbitrary guesses—and tune crowd behavior with a single variable. As a work-in-progress, future research will focus on experimental validation, testing if human workers (who may not always act as perfectly rational agents) respond to these game-theoretic incentives as predicted in real-world activity recognition and image labeling tasks.

Summary updated

Abstract

One common problem plaguing crowdsourcing tasks is tuning the set of worker responses: Depending on task requirements, requesters may want a large set of rich and varied worker responses (typically in subjective evaluation tasks) or a more convergent response-set (typically for more objective tasks such as fact-checking). This problem is especially salient in tasks that combine workers’ responses to present a single output: Divergence in these settings could either add richness and complexity to the unified answer, or noise. In this paper we present HiveMind, a system of methods that allow requesters to tune different levels of convergence in worker participation for different tasks simply by adjusting the value of one variable.
BibTeX
@inproceedings{singh2013hivemind,
  author = {Preetjot Singh and Walter S. Lasecki and Paulo Barelli and Jeffrey P. Bigham},
  title = {{HiveMind: Tuning Crowd Response with a Single Value}},
  booktitle = {HCOMP (Works in Progress / Demos)},
  year = {2013},
  url = {https://openreview.net/forum?id=dyo_ZIIQeBx},
}

Loading related content...

Figure