Un turc mécanique pour les ressources linguistiques : critique de la myriadisation du travail parcellisé

Benoît Sagot, Karën Fort, Gilles Adda, Joseph Mariani, Bernard LangCalifornia 2011

TL;DR

This paper critically examines the use of Amazon Mechanical Turk (MTurk) for creating linguistic resources, arguing that its perceived low cost and high speed mask significant issues with quality, worker status, and ethics. The authors propose that while MTurk offers a paradigm shift, alternative, more ethical, and potentially higher-quality methods for resource development exist and should be prioritized.

What Problem Does This Paper Address?

The paper addresses the growing trend of using crowdsourcing platforms like Amazon Mechanical Turk (MTurk) for Natural Language Processing (NLP) tasks, particularly for creating linguistic resources. While the prevailing narrative in NLP research suggests MTurk offers high-quality resources quickly and cheaply, the authors question this assumption. They highlight a gap between the perceived benefits and the actual implications, particularly concerning the quality of the resulting data, the economic viability and ethical treatment of the workers ("Turkers"), and the legal ramifications of this type of labor. The paper aims to provide researchers with a more informed perspective to make conscious choices about using MTurk and to encourage the development of more responsible resource creation practices.

Methods

The paper is primarily a position paper and a critical review, rather than a paper presenting novel algorithms or datasets. The authors employ a qualitative analysis of existing literature and anecdotal evidence related to MTurk and crowdsourcing in NLP. They:

  • Critically analyze the discourse: They challenge the common claims about MTurk's cost-effectiveness and speed by examining the hidden costs, quality control issues, and ethical considerations.
  • Review existing literature: They cite numerous studies that have used MTurk, analyzing their findings and the authors' own interpretations of the results.
  • Examine worker demographics and motivations: They discuss survey data on who the Turkers are, their motivations for participating, and how they perceive their work.
  • Discuss legal and ethical frameworks: They explore the implications of MTurk for labor laws, worker rights, and intellectual property.
  • Propose alternatives: They review and discuss alternative methods for linguistic resource development, including unsupervised and semi-supervised learning, re-use of existing resources, and collaborative development platforms.

Datasets

The paper does not introduce new datasets. Instead, it discusses the use of MTurk for creating various types of linguistic resources, such as:

  • Annotated corpora
  • Named entity recognition data
  • Speech and language data
  • Word alignment data
  • Summarization evaluation data
  • Lexical chains
  • Noun compound interpretation data

The paper references numerous studies that have used MTurk to generate these types of data, but it does not detail specific datasets used for training or evaluation within this paper itself.

Key Insights and Ideas

  • The "low cost" of MTurk is often an illusion: Hidden costs related to quality control, task design, and post-processing are frequently overlooked, making the true cost higher than initially perceived.
  • Quality is not guaranteed: While MTurk can produce usable data, the quality is highly variable and often insufficient for complex tasks, requiring significant expert validation or post-processing.
  • Worker status is ethically ambiguous: The paper argues that many MTurk tasks blur the lines between hobby, freelance work, and undeclared labor, raising serious ethical and legal concerns regarding worker rights and fair compensation.
  • The "hobbyist" narrative is misleading: While some Turkers may treat it as a hobby, a significant portion relies on MTurk for income, making the low pay rates exploitative.
  • Legal frameworks are ill-equipped: Existing labor laws, particularly in France, struggle to categorize MTurk work, potentially leading to undeclared labor and tax evasion.
  • Alternatives exist and are often superior: The paper advocates for a range of alternatives, including unsupervised learning, leveraging existing resources, and more ethical crowdsourcing platforms, which can achieve comparable or better results without the ethical drawbacks.
  • The "MTurk norm" is a dangerous precedent: The authors warn that the widespread adoption of MTurk could devalue linguistic resources and pressure funding agencies to accept artificially low costs, undermining fair labor practices in the long run.

Key Results

The paper does not present quantitative results from experiments. Instead, its "results" are qualitative findings derived from its critical analysis:

  • Increased publication trend: The paper notes a significant increase in publications using MTurk in NLP conferences, indicating its growing popularity.
  • Worker demographics and motivations: Survey data suggests that while financial motivation is significant for some (especially in India), many Turkers also cite interest in tasks or simply "killing time." However, a substantial percentage (around 20%) rely on MTurk for their primary income.
  • Cost variability: The paper highlights instances where MTurk tasks became significantly more expensive than initially planned due to a lack of qualified workers or the need to increase pay to attract participants.
  • Quality limitations: The paper cites studies showing that non-expert annotators on MTurk often produce lower quality results for complex tasks compared to experts, and that even for simpler tasks, quality can vary significantly.
  • Legal status ambiguity: The paper concludes that MTurk's classification of workers as independent contractors is legally questionable, especially in jurisdictions like France, and that many Turkers likely do not declare income or pay social contributions.

Limitations and Future Work

Limitations:

  • Lack of direct experimentation: The paper relies heavily on secondary analysis of existing literature and does not present its own empirical studies or comparisons of MTurk against alternatives.
  • Generalizability of findings: While the paper focuses on NLP, the broader implications for other fields using crowdsourcing are not deeply explored.
  • Defining "work": The paper acknowledges that the definition of "work" and "fair compensation" is complex and can vary, making definitive judgments challenging.

Future Work:

  • Systematic comparison: The authors suggest that a systematic comparison between MTurk and state-of-the-art algorithms for various NLP tasks is needed.
  • Development of ethical labels: They propose the creation of quality and ethical labels for linguistic resources, awarded by professional associations, to guide funding and selection processes.
  • Exploration of alternative platforms: Further research into and development of more ethical and effective crowdsourcing platforms is encouraged.
  • Legal clarity: The need for clearer legal frameworks to govern crowdsourced work is emphasized.

Summary updated

Abstract

Misleading symptoms were responsible for failure to make the diagnosis of symptomatic abdominal aortic aneurysm in 15 patients. The presenting complaints appeared to be specific for other diseases, such as genitourinary disease, diverticulitis, intra-abdominal neoplasm and functional large intestinal disorders. A correct diagnosis was ultimately made in 12 patients and aneurysmectomy was performed. In three patients, who died of ruptured aneurysm, the diagnosis was not made until postmortem examination. An awareness of the atypical symptoms of aneurysms, careful physical examination and appropriate x-ray studies will lead to the diagnosis of symptomatic aortic aneurysms. Early resection will result in a lower mortality rate.
BibTeX
@inproceedings{sagot2011un,
  author = {Benoît Sagot and Karën Fort and Gilles Adda and Joseph Mariani and Bernard Lang},
  title = {{Un turc mécanique pour les ressources linguistiques : critique de la myriadisation du travail parcellisé}},
  booktitle = {California},
  year = {2011},
}
Figure