Cost Optimization in Crowdsourcing Translation: Low cost translations made even cheaper

Mingkun Gao, Wei Xu, Chris Callison-BurchNAACL 2015 · DOI

Abstract

High-quality data is a significant bottleneck in Natural Language Processing, particularly for Statistical Machine Translation (SMT) which requires millions of parallel sentence pairs. While crowdsourcing offers a cheaper alternative to professional translation, the inherent variability in worker quality often necessitates expensive redundancy. This paper introduces two strategies to optimize costs: dynamic stopping, which halts data collection once a "good enough" translation is received, and rapid worker filtering, which identifies high-performing workers early. Experimental results on Urdu-English translation tasks demonstrate that these methods can reduce costs by 50% without compromising the quality of the resulting translation models.

Introduction

The development of modern translation models is heavily dependent on large-scale, high-quality datasets. Traditionally, professional translation services have been the gold standard, but at approximately $6.00 per sentence, the cost is prohibitive for the scale required by modern SMT systems. Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) have emerged as a viable alternative, offering costs as low as $0.10 per sentence. However, the quality of crowd-generated data is inconsistent. To mitigate this, researchers often collect multiple redundant translations per sentence, which, while effective, significantly inflates the total cost. This research explores whether professional-level quality can be achieved with fewer redundant translations through intelligent optimization.

Methods for Cost Optimization

The researchers proposed a two-pronged approach to reduce redundancy costs:

1. Dynamic Stopping (The "Good Enough" Method)

Instead of a fixed redundancy (e.g., four translations per sentence), the system employs a linear regression model to estimate the quality of each incoming translation in real-time. Quality is measured using the BLEU score. If a translation's estimated quality meets a predefined threshold ($\theta$), the system stops requesting further translations for that sentence and moves resources to the next.

2. Rapid Worker Filtering

A key observation in crowdsourcing is that worker performance is remarkably consistent. The authors developed a method to rank workers based on a small initial sample of their work (as few as 20 sentences). By identifying and retaining only the top-tier workers early in the process, the system avoids paying for low-quality output from unreliable sources.

Technical Highlight: Alignment-Based Quality Estimation

To accurately estimate translation quality without a reference, the authors introduced a "bilingual feature" leveraging IBM Model 1. This feature calculates the alignment probability between words in the source sentence and the provided translation.

Figure 1: Example of word alignment between an Urdu source sentence and English translations, used to calculate alignment probabilities for quality estimation.

If the alignment between the source and target words is poor, it serves as a strong indicator of a low-quality or spammy translation. This automated alignment acts as a proxy for human quality, allowing the system to make informed decisions about when to stop collecting data.

Key Insights: Worker Consistency

The effectiveness of the worker filtering strategy rests on the high correlation between a worker's early performance and their long-term quality. The study found that a ranking based on just the first 20 sentences had a 0.94 correlation with a "gold standard" ranking derived from the workers' entire history.

Figure 6: Correlation between worker rankings based on the first 20 sentences versus their overall 'gold' ranking. The high correlation supports rapid filtering of low-quality workers.

Key Results

The combined strategies yielded significant improvements in efficiency:

  • 50% Cost Reduction: The system achieved a BLEU score of 38.05 (comparable to professional quality) using an average of only 2.2 translations per sentence, a nearly 50% reduction from the baseline of 4.0.
  • High-Accuracy Ranking: The quality estimation model achieved a 0.95 correlation with gold-standard worker rankings when all features were utilized.
  • Performance Stability: Even when the worker pool was aggressively filtered (removing 75% of workers), the system's performance remained stable, losing only 0.55 BLEU points compared to a scenario with perfect worker knowledge.

Limitations and Future Work

The method currently requires a small amount of professional "gold standard" data (about 10%) for initial calibration. While the features used (like sentence length and alignment) are general, the model's weights might need re-tuning for different language pairs beyond Urdu-to-English. The authors suggest this framework could be applied to other crowdsourced tasks, such as image labeling or sentiment analysis, where worker consistency can be leveraged to optimize costs.

Summary updated

BibTeX
@inproceedings{gao2015cost,
  author = {Mingkun Gao and Wei Xu and Chris Callison-Burch},
  title = {{Cost Optimization in Crowdsourcing Translation: Low cost translations made even cheaper}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1072},
}
Figure