Crowdsourcing for language resource development: Critical analysis of amazon mechanical turk overpowering use

Gilles Adda, Benoît Sagot, Karën Fort, Joseph Mariani2011

TL;DR

This paper critically examines the prevailing optimistic view of Amazon Mechanical Turk (MTurk) for language resource development. It argues that while MTurk offers cost and speed advantages, it presents significant challenges regarding quality, worker ethics, and labor rights, urging researchers to consider alternatives.

What Problem Does This Paper Address?

The paper addresses the growing reliance on crowdsourcing platforms like Amazon Mechanical Turk (MTurk) for creating language resources (e.g., annotated corpora, transcriptions). The common perception is that MTurk allows for rapid and inexpensive development of high-quality resources by leveraging a large pool of online workers who treat it as a hobby or a way to earn extra money. However, the authors question this idealized view, highlighting potential issues with data quality, the true cost-effectiveness, and the ethical implications for the workers. They aim to provide researchers with a more balanced perspective to make informed decisions about using such platforms.

Methods

This paper is primarily a position paper and a critical analysis, rather than a presentation of novel algorithms or experimental results. The authors synthesize existing research, survey data, and anecdotal evidence to build their argument. They analyze the "legend" of MTurk's efficiency and low cost against the "truth" by examining:

  • Worker Motivation and Status: Investigating whether MTurk workers are truly hobbyists or if a significant portion are engaged in labor, with implications for wages and rights.
  • Cost Analysis: Moving beyond just the per-task payment to include hidden costs like interface development, spammer management, and quality validation.
  • Quality Assessment: Evaluating the actual quality of data produced by MTurk workers, especially for complex tasks, and comparing it to expert annotations.
  • Ethical and Legal Considerations: Discussing the lack of worker protections, the ambiguity of the employer-employee relationship, and the potential for exploitation.

They also review existing and proposed alternatives to MTurk, including unsupervised/semi-supervised learning techniques, reusing existing resources, and other collaborative or ethical crowdsourcing platforms.

Datasets

The paper does not introduce new datasets or benchmarks. Instead, it references numerous studies that have used MTurk for various language processing tasks, such as:

  • Transcription and translation
  • Named-entity recognition (NER)
  • Part-of-speech (POS) tagging
  • Word-sense disambiguation
  • Summarization system evaluation

The authors cite studies that have used MTurk data to train and evaluate statistical translation/transcription systems, and others that have highlighted quality issues with MTurk-generated data.

Key Insights and Ideas

  • MTurk is not just a hobby: A significant portion of MTurk workers rely on it for income, with some even considering it their primary source of earnings. This reframes the activity as labor, raising ethical concerns about low wages and lack of worker protections.
  • Hidden costs can negate apparent savings: While MTurk tasks might seem cheap, the overall cost can increase significantly due to the effort required for interface development, quality control, validation, and dealing with spam or low-quality submissions.
  • Quality is not guaranteed, especially for complex tasks: While MTurk can be sufficient for some simple tasks, its effectiveness diminishes for more complex linguistic annotations where expert knowledge is crucial. The quality can be highly variable and often requires extensive post-processing.
  • Lack of worker rights is a major concern: MTurk's design obscures the relationship between requesters and workers, denying basic labor rights like unionization and recourse against unfair practices.
  • Alternatives exist and should be explored: The paper strongly advocates for exploring unsupervised/semi-supervised learning, leveraging existing resources, and investigating more ethical crowdsourcing platforms or collaborative approaches.

Key Results

The paper presents qualitative findings and synthesizes results from other studies rather than reporting new quantitative outcomes. Key findings include:

  • Worker Demographics: Studies cited indicate that a substantial percentage of MTurk workers desire to make money, with a notable proportion relying on it for basic income or as a primary source.
  • Wage Discrepancies: Wages on MTurk can be very low, sometimes below $2 per hour, though specific tasks requiring specialized skills (e.g., Korean speech transcription) can command higher rates.
  • Quality Variability: While some papers report sufficient quality for training MTurk-based systems, others highlight significant quality problems, particularly for complex tasks, where expert annotators consistently outperform MTurk workers.
  • Cost Factors: The paper emphasizes that costs associated with interface development, validation, and correction can be substantial, potentially offsetting the low per-task payment.

Limitations and Future Work

Limitations:

  • Position Paper Nature: The paper is a critical analysis and synthesis, not an empirical study presenting new data or experimental results. Its conclusions are based on interpreting existing literature and survey data.
  • Generalization Challenges: The authors acknowledge that MTurk encompasses a wide variety of tasks and worker motivations, making it difficult to draw absolute conclusions that apply universally.
  • Interface Issues: The paper notes that interface limitations are a significant drawback, but these are often generic to crowdsourcing systems.

Future Work/Suggestions:

  • Inform Researchers: The primary goal is to equip researchers with a comprehensive understanding of MTurk's pros and cons to make informed decisions.
  • Seek Support for Alternatives: The authors call for help from funding agencies and scientific associations to develop and promote alternative, ethical, and high-quality language resource development methods.
  • Develop Practical Solutions: Propose practical and organizational solutions to improve crowdsourced resource development while mitigating ethical and legal risks.
  • Advocate for Ethical Crowdsourcing: Encourage the development of platforms that ensure fair wages, worker rights, and transparency, potentially through collaboration with professional associations like ACL, ISCA, and ELRA.
  • Explore Optimized Expert Annotation: Suggest that even costly expert annotation can be made more efficient and cost-effective through optimized workflows and dedicated interfaces.

Summary updated

Abstract

This article is a position paper about crowdsourced microworking systems and especially Amazon Mechanical Turk, the use of which has been steadily growing in language processing in the past few years. According to the mainstream opinion expressed in the articles of the domain, this type of on-line working platforms allows to develop very quickly all sorts of quality language resources, for a very low price, by people doing that as a hobby or wanting some extra cash. We shall demonstrate here that the situation is far from being that ideal, be it from the point of view of quality, price, workers ’ status or ethics and bring back to mind already existing or proposed alternatives. Our goal here is threefold: 1- to inform researchers, so that they can make their own choices with all the elements of the reflection in mind, 2- to ask for help from funding agencies and scientific associations, and develop alternatives, 3- to propose practical and organizational solutions in order to improve new language resources development, while limiting the risks of ethical and legal issues without letting go price or quality.
BibTeX
@misc{adda2011crowdsourcing,
  author = {Gilles Adda and Benoît Sagot and Karën Fort and Joseph Mariani},
  title = {{Crowdsourcing for language resource development: Critical analysis of amazon mechanical turk overpowering use}},
  year = {2011},
}

Loading related content...

Figure