Experiments on Motivational Feedback for Crowdsourced Workers

Tak Yeon Lee, Casey Dugan, Werner Geyer, Tristan Ratchford, Jamie C. Rasmussen, N. Sadat Shami, Stela Lupushor

Abstract

This paper examines the relationship between motivational design and its longitudinal effects on crowdsourcing systems. In the context of a company internal web site that crowdsources the identification of Twitter accounts owned by company employees, we designed and investigated the effects of various motivational features including individual / social achievements and gamification. Our 6-month experiment with 437 users allowed us to compare the features in terms of both quantity and quality of the work produced by participants over time. While we found that gamification can increase workers’ motivation overall, the combination of motivational features also matters. Specifically, gamified social achievement is the best performing design over a longer period of time. Mixing individual and social achievements turns out to be less effective and can even encourage users to game the system.

Introduction

Crowdsourcing platforms face a dual challenge: maximizing worker output while maintaining high data quality. While financial incentives are common in platforms like Amazon Mechanical Turk, internal enterprise systems often rely on "soft" motivators such as fun, reputation, or altruism. This research addresses a critical gap in UI design: understanding how different motivational elements—such as personal progress bars versus social leaderboards—interact over extended periods. Does adding more "game-like" features always help, or is there a point of diminishing returns where users prioritize points over task accuracy?

Methodology

The study utilized IBMersWhoTweet, an internal tool designed to identify the Twitter accounts of IBM employees. The workflow involved two main steps: creating a pool of potential accounts via API keyword searches and then using a crowdsourced identification task to verify them against internal directories.

The two-step process for identifying IBM employees on Twitter: (1) keyword-based account pooling and (2) crowdsourced identification.

Researchers implemented a 2x3 factorial design to test seven feedback environments:

  1. Individual Achievement: Displaying personal identification counts.
  2. Social Achievement: Displaying community progress and recent contributors.
  3. Gamification: Adding levels/badges (individual) or leaderboards (social).
  4. Combinations: Testing both individual and social features together, in both gamified and non-gamified versions.

The experiment tracked 437 active participants over six months, measuring both the quantity and accuracy of classifications. The task pool consisted of 7,016 potentially relevant accounts, resulting in 4,154 total classifications.

Key Insights

  • The "More is Less" Paradox: Surprisingly, the baseline group (no feedback) outperformed all non-gamified feedback designs. This suggests that poorly designed feedback can distract from the intrinsic motivation of the task itself.
  • Social vs. Individual Longevity: While gamified individual feedback (badges) is effective for initial engagement, social gamification (leaderboards) is the "marathon runner" of motivation, showing significantly higher retention rates after months of use.
  • The Threshold of Gaming: Gamification generally improved accuracy (79% vs 56% in non-gamified groups). However, over-saturating the UI with every possible feature (Individual + Social + Gamification) led to a drop in quality as users began "gaming the system" to climb ranks, sacrificing quality for speed.
  • Power Law Behavior: Worker activity followed the "90-9-1" rule, where a tiny fraction of "super-workers" performed the vast majority of tasks. Social gamification was particularly effective at motivating these high-performers.

Results and Impact

The study demonstrated that gamified social design could motivate workers to complete ten times more tasks than non-gamified social designs. Gamified groups were responsible for 73.7% of all work completed. The final output of this effort was a verified feed of tweets from identified IBMers, providing a valuable internal resource.

The final application: a curated feed of tweets from verified IBM employees.

Conclusion

Gamification is a potent tool for crowdsourcing, but its implementation requires careful balance. Gamified social achievement stands out as the most effective driver for long-term engagement. Future work should explore adaptive UIs that evolve with the user's journey, starting with individual hooks and transitioning to social motivators to maintain longitudinal productivity without compromising data quality.

Summary updated

Abstract

This paper examines the relationship between motivational design and its longitudinal effects on crowdsourcing systems. In the context of a company internal web site that crowdsources the identification of Twitter accounts owned by company employees, we designed and investigated the effects of various motivational features including individual / social achievements and gamification. Our 6-month experiment with 437 users allowed us to compare the features in terms of both quantity and quality of the work produced by participants over time. While we found that gamification can increase workers’ motivation overall, the combination of motivational features also matters. Specifically, gamified social achievement is the best performing design over a longer period of time. Mixing individual and social achievements turns out to be less effective and can even encourage users to game the system.
BibTeX
@inproceedings{lee2021experiments,
  author = {Tak Yeon Lee and Casey Dugan and Werner Geyer and Tristan Ratchford and Jamie C. Rasmussen and N. Sadat Shami and Stela Lupushor},
  title = {{Experiments on Motivational Feedback for Crowdsourced Workers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v7i1.14428},
}
Figure