Abstract
High-quality data is a significant bottleneck in Natural Language Processing, particularly for Statistical Machine Translation (SMT) which requires millions of parallel sentence pairs. While crowdsourcing offers a cheaper alternative to professional translation, the inherent variability in worker quality often necessitates expensive redundancy. This paper introduces two strategies to optimize costs: dynamic stopping, which halts data collection once a "good enough" translation is received, and rapid worker filtering, which identifies high-performing workers early. Experimental results on Urdu-English translation tasks demonstrate that these methods can reduce costs by 50% without compromising the quality of the resulting translation models.
Introduction
The development of modern translation models is heavily dependent on large-scale, high-quality datasets. Traditionally, professional translation services have been the gold standard, but at approximately $6.00 per sentence, the cost is prohibitive for the scale required by modern SMT systems. Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) have emerged as a viable alternative, offering costs as low as $0.10 per sentence. However, the quality of crowd-generated data is inconsistent. To mitigate this, researchers often collect multiple redundant translations per sentence, which, while effective, significantly inflates the total cost. This research explores whether professional-level quality can be achieved with fewer redundant translations through intelligent optimization.
Methods for Cost Optimization
The researchers proposed a two-pronged approach to reduce redundancy costs:
1. Dynamic Stopping (The "Good Enough" Method)
Instead of a fixed redundancy (e.g., four translations per sentence), the system employs a linear regression model to estimate the quality of each incoming translation in real-time. Quality is measured using the BLEU score. If a translation's estimated quality meets a predefined threshold ($\theta$), the system stops requesting further translations for that sentence and moves resources to the next.
2. Rapid Worker Filtering
A key observation in crowdsourcing is that worker performance is remarkably consistent. The authors developed a method to rank workers based on a small initial sample of their work (as few as 20 sentences). By identifying and retaining only the top-tier workers early in the process, the system avoids paying for low-quality output from unreliable sources.
Technical Highlight: Alignment-Based Quality Estimation
To accurately estimate translation quality without a reference, the authors introduced a "bilingual feature" leveraging IBM Model 1. This feature calculates the alignment probability between words in the source sentence and the provided translation.
If the alignment between the source and target words is poor, it serves as a strong indicator of a low-quality or spammy translation. This automated alignment acts as a proxy for human quality, allowing the system to make informed decisions about when to stop collecting data.
Key Insights: Worker Consistency
The effectiveness of the worker filtering strategy rests on the high correlation between a worker's early performance and their long-term quality. The study found that a ranking based on just the first 20 sentences had a 0.94 correlation with a "gold standard" ranking derived from the workers' entire history.
Key Results
The combined strategies yielded significant improvements in efficiency:
- 50% Cost Reduction: The system achieved a BLEU score of 38.05 (comparable to professional quality) using an average of only 2.2 translations per sentence, a nearly 50% reduction from the baseline of 4.0.
- High-Accuracy Ranking: The quality estimation model achieved a 0.95 correlation with gold-standard worker rankings when all features were utilized.
- Performance Stability: Even when the worker pool was aggressively filtered (removing 75% of workers), the system's performance remained stable, losing only 0.55 BLEU points compared to a scenario with perfect worker knowledge.
Limitations and Future Work
The method currently requires a small amount of professional "gold standard" data (about 10%) for initial calibration. While the features used (like sentence length and alignment) are general, the model's weights might need re-tuning for different language pairs beyond Urdu-to-English. The authors suggest this framework could be applied to other crowdsourced tasks, such as image labeling or sentiment analysis, where worker consistency can be leveraged to optimize costs.
Summary updated