Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers

Saiph Savage, Chun-Wei Chiang, Susumu Saito, Carlos Toxtli, Jeffrey P. Bigham

TL;DR

While transparency tools for crowd work exist, most novice "Turkers" still earn well below the minimum wage because they lack a strategy to interpret the data. This paper identifies a specific decision-making hierarchy used by "Super Turkers" (high earners) and demonstrates that when novices follow this simple playbook, they can significantly increase their hourly earnings, even when the broader market is lean.

What Problem Does This Paper Address?

Crowd markets like Amazon Mechanical Turk (MTurk) are notorious for "invisible labor" and low pay, with many workers struggling to reach the U.S. minimum wage of $7.25/hour. While the community has developed transparency tools like Turkopticon and TurkerView to rate requesters, having data isn't the same as having a strategy.

Novices often feel overwhelmed by the various metrics (generosity, promptness, fairness, etc.) and end up picking tasks inefficiently. The authors aim to bridge this "guidance gap" by computationally identifying the heuristics used by the platform’s top earners—the "Super Turkers"—and testing if those strategies can be successfully transferred to beginners.

Methods

The researchers employed a two-phase approach:

  1. Strategy Extraction: They surveyed 100 Super Turkers (defined as having completed >10,000 tasks and earning above minimum wage). Using the Analytic Hierarchy Process (AHP) and a Longest Common Subsequence (LCS) algorithm, they distilled the experts' complex workflows into a simple, three-step hierarchical decision tree.
  2. Field Experiment: They recruited 100 novice workers and conducted a 12-day study using a custom Chrome plugin. The plugin tracked "real" hourly wages by measuring the exact time spent searching for and completing tasks. They split the novices into a control group (using tools as they saw fit) and an experimental group (instructed to follow the Super Turker strategy).

Datasets

  • Super Turker Survey: Responses from 68 validated high-earning workers detailing their tool usage and priority metrics.
  • Novice Field Study Data: A dataset of over 25,000 tasks performed by 100 novices, involving 1,394 unique requesters. This includes metadata on HIT (Human Intelligence Task) rewards, requester reputation, and precise timestamps for worker activity.

Key Insights and Ideas

  • The "Super Turker" Heuristic: The study revealed a clear priority list for selecting work:
  • Prioritize tasks where TurkerView predicts an hourly wage >$8.29/h.
  • If that data is missing, only work for requesters with a Turkopticon fairness score >3.69.
  • If no reputation data exists, only accept tasks with a flat reward >$0.23.
  • The Search Time Trade-off: Identifying "Super HITs" (tasks meeting the criteria) takes significantly longer than picking tasks at random. Novices spent a median of 61 seconds searching for a Super HIT compared to just 9 seconds for a normal one.
  • The "Fairness" Proxy: When wage data is unavailable, Super Turkers rely heavily on "Fairness" ratings. This is a defensive strategy to avoid "algorithmic cruelty"—situations where a requester might arbitrarily reject work, leading to zero pay and a damaged worker reputation.
  • Resilience to Market Fluctuations: During the study, the general pool of available tasks on MTurk decreased in quality. While the control group's wages plummeted, the novices following the Super Turker strategy were able to maintain their wage levels.

Key Results

  • Higher Ceiling: Individual "Super HITs" identified by the criteria paid a median of $7.04/h**, compared to just **$3.27/h for non-Super HITs.
  • Net Gains: Even after accounting for the extra unpaid time spent searching for better work, the median hourly wage for those following the strategy was $4.12/h**, nearly double the **$2.13/h earned by those who didn't.
  • Statistical Significance: A Mann-Whitney U test confirmed a significant difference ($p = 0.006$) in how median hourly wages varied between the control and experimental groups during the test phase.

Limitations and Future Work

  • The Compliance Hurdle: About half of the novices in the experimental group failed to follow the instructions consistently. The authors suggest that future tools should not just provide data, but perhaps use "on-the-job tutorials" or more assertive interfaces to guide behavior.
  • Qualification Barriers: Many high-paying tasks require specific "qualifications" (MTurk certifications). The study did not track these, and novices may have been barred from the best tasks regardless of their strategy.
  • Ecosystem Volatility: The study highlights how dependent workers are on the daily "task pool." Future work could explore how these strategies hold up across different crowd platforms like Prolific or Upwork.
  • Privacy: The researchers noted the ethical tension in tracking worker behavior and emphasized the need for privacy-preserving transparency tools.

Summary updated

Abstract

Crowd markets have traditionally limited workers by not providing transparency information concerning which tasks pay fairly or which requesters are unreliable. Researchers believe that a key reason why crowd workers earn low wages is due to this lack of transparency. As a result, tools have been developed to provide more transparency within crowd markets to help workers. However, while most workers use these tools, they still earn less than minimum wage. We argue that the missing element is guidance on how to use transparency information. In this paper, we explore how novice workers can improve their earnings by following the transparency criteria of Super Turkers, i.e., crowd workers who earn higher salaries on Amazon Mechanical Turk (MTurk). We believe that Super Turkers have developed effective processes for using transparency information. Therefore, by having novices follow a Super Turker criteria (one that is simple and popular among Super Turkers), we can help novices increase their wages. For this purpose, we: (i) conducted a survey and data analysis to computationally identify a simple yet common criteria that Super Turkers use for handling transparency tools; (ii) deployed a two-week field experiment with novices who followed this Super Turker criteria to find better work on MTurk. Novices in our study viewed over 25,000 tasks by 1,394 requesters. We found that novices who utilized this Super Turkers' criteria earned better wages than other novices. Our results highlight that tool development to support crowd workers should be paired with educational opportunities that teach workers how to effectively use the tools and their related metrics (e.g., transparency values). We finish with design recommendations for empowering crowd workers to earn higher salaries.
BibTeX
@inproceedings{savage2020becoming,
  author = {Saiph Savage and Chun-Wei Chiang and Susumu Saito and Carlos Toxtli and Jeffrey P. Bigham},
  title = {{Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers}},
  booktitle = {The Web Conference},
  year = {2020},
  doi = {10.1145/3366423.3380200},
}

Loading related content...

Figure