TL;DR
This paper investigates "qualification labor" – the unpaid work crowd workers must do to gain qualifications for better-paid tasks. The authors find that common qualification thresholds (like 5,000 approved tasks) can require months of low-paid work, and propose and test alternative, lower thresholds that maintain data quality.
What Problem Does This Paper Address?
Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) are widely used for tasks like data annotation. However, many crowd workers earn wages far below the minimum wage. To ensure data quality, researchers often restrict tasks to workers who have a certain number of "Approved HITs" (Human Intelligence Tasks). While this can lead to higher wages for qualified workers, it creates a significant barrier: workers must first complete a large volume of low-paid, potentially unpleasant tasks just to become eligible for better-paid work. This "qualification labor" is unpaid and can take months, effectively subsidizing researchers at the expense of workers. The paper aims to quantify this issue and explore alternative qualification strategies that balance worker fairness with researcher needs for quality data.
Methods
The authors first analyze existing literature and online discussions (forums, Reddit) to understand common qualification norms and the estimated time commitment involved. They then conduct an empirical study on two NLP tasks: coreference resolution and sentiment analysis. For these tasks, they recruit workers and group them based on different combinations of "Approved HITs" and "Percentage of HITs Accepted" qualifications. They then measure the quality of the annotations produced by these different groups. They also explore alternative qualification methods, such as screening questions and controlled crowdsourcing.
Datasets
The paper uses data from two NLP tasks:
- Coreference Resolution: Based on the Ontonotes dataset, using noun phrases identified by the Allen NLP parser.
- Sentiment Analysis: Based on the Stanford Sentiment Treebank.
The study itself involves recruiting workers from Amazon Mechanical Turk to perform these tasks.
Key Insights and Ideas
- Qualification Labor is a Real Burden: The paper quantifies that achieving common qualification milestones (like 5,000 approved HITs) can take over two months of full-time, poorly paid work.
- Common Thresholds are High: Many sources (MTurk blog, forums, academic papers) suggest thresholds of 5,000 or 10,000 approved HITs, which are substantial.
- Lower Thresholds Can Work: The empirical study suggests that for both coreference resolution and sentiment analysis, high-quality annotations can be achieved with lower qualification thresholds, particularly by focusing on the "Percentage of HITs Accepted."
- Task Dependency Matters: The optimal qualification strategy is not one-size-fits-all; it depends on the specific task's complexity and the required annotation quality.
- Worker Self-Selection: Higher qualification requirements can lead to more workers returning tasks without completing them, suggesting they are self-selecting out due to perceived difficulty or unfairness.
Key Results
- For coreference resolution, workers with an acceptance percentage below 99% consistently produced poor results. However, with an acceptance percentage of 99-100%, higher approved HITs thresholds did not necessarily lead to better quality, and some workers self-selected out.
- For sentiment analysis, the trend of needing a high acceptance percentage (above 99%) was repeated. However, the impact of the "Approved HITs" threshold showed a U-shaped pattern, indicating task dependency.
- The study suggests that for both tasks, it's possible to reduce the burden on workers by using lower qualification thresholds without sacrificing data quality.
Limitations and Future Work
- Sample Size: The study's experimental groups had relatively small sample sizes (12-15 workers per group), which the authors acknowledge.
- Worker Bias: The filtering process for qualification can potentially create a biased sample of workers, which might be problematic for subjective tasks.
- Future Work: The authors recommend further research into task-specific calibration of qualification thresholds. They also highlight controlled crowdsourcing (where an initial task identifies qualified workers for a subsequent, full task) as a promising method to reduce qualification labor while maintaining quality. They suggest exploring alternative qualification methods beyond just "Approved HITs" and "Percentage Accepted."
Summary updated