TL;DR
This paper systematically compares the use of professional vs. crowdsourced (Amazon Mechanical Turk) translations for training, tuning, and evaluating Machine Translation (MT) systems. They found that crowdsourced translations can achieve comparable MT performance at a fraction of the cost, and that using a second crowdsourced reference translation improves evaluation accuracy.
What Problem Does This Paper Address?
High-quality parallel data (source sentences paired with their translations) is crucial for training effective Machine Translation (MT) systems. However, creating this data, especially reference translations, is expensive and time-consuming when relying on professional translators. This paper investigates whether crowdsourcing platforms like Amazon Mechanical Turk (MTurk) can provide a viable, cost-effective alternative for generating these essential translation resources, and how their use impacts MT system performance.
Methods
The authors employ a standard statistical MT (SMT) pipeline, specifically a string-to-dependency-tree hierarchical model. They leverage existing sentence alignment tools (GIZA++) and a dependency tree language model. The novelty lies not in the MT architecture itself, but in how they generate and utilize the training, tuning, and testing data. They compare MT systems trained on data translated by professionals versus those translated by MTurk workers. They also explore the impact of using multiple reference translations (both professional and MTurk) for tuning and scoring.
Datasets
The primary dataset used is a collection of Arabic-English parallel corpora from the DARPA BOLT program. This data, originally collected from web discussion forums, was professionally translated into English. The authors then obtained alternative MTurk translations for a significant portion of this data. They also incorporated 10 million words of parallel data from the NIST MT 2012 corpora (mostly news) to investigate domain adaptation scenarios.
Key Insights and Ideas
- Crowdsourcing as a Cost-Effective Alternative: MTurk translations can be obtained at a significantly lower cost (up to 1/10th the price of professional translations) and much faster.
- Comparable MT Performance: MT systems trained on MTurk translations achieve performance comparable to those trained on professional translations, especially when considering the cost savings.
- Improved Evaluation with Multiple References: Using a second, crowdsourced reference translation for the development set leads to more reliable evaluation of MT output and better parameter tuning.
- Domain Adaptation Benefits: In scenarios where out-of-domain data (like news) is already available, crowdsourced translations for the in-domain data can be used with minimal loss in MT performance.
- Length Mismatch Nuance: MTurk translations tend to be shorter than professional ones, which can affect tuning and evaluation if not accounted for.
Key Results
- MTurk translations can be obtained at 1/10th the price of professional translations.
- Training MT systems on MTurk references yields the same performance as professional translations at 20% of the cost.
- Using MTurk translations for training data, the cost ratio for achieving the same MT performance is 5x (due to needing twice the amount of MTurk data).
- Scoring MT output using both professional and MTurk references increases BLEU scores by over 5 points.
- Tuning system parameters using both professional and MTurk references yields a gain of 2.5 to 2.7 BLEU points.
- When using MTurk translations for tuning, the output length is shortened by around 5% on average.
Limitations and Future Work
The paper acknowledges that MTurk translations can be less fluent and shorter than professional translations. This length mismatch can impact tuning and evaluation, requiring careful consideration. The authors suggest that future work could explore more sophisticated quality control mechanisms for crowdsourced data and investigate the use of crowdsourced translations in other NLP tasks beyond MT.
Summary updated