Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators

Omar F. Zaidan, Chris Callison-BurchNAACL 2010

TL;DR

This paper tackles the challenge of evaluating machine translation (MT) quality using Human-Targeted Translation Edit Rate (HTER), a metric that requires human post-editing. They propose using untrained Amazon Mechanical Turk (AMT) workers to generate these post-edits, demonstrating that this crowdsourced data can predict document rankings significantly better than traditional automatic metrics, and at a low cost.

What Problem Does This Paper Address?

Evaluating machine translation (MT) systems has traditionally relied on human judgment, which is accurate but prohibitively expensive and time-consuming for rapid development cycles. Automatic metrics like BLEU emerged to fill this gap, allowing researchers to track progress and assess the impact of small system changes. However, for certain critical evaluations, like those in the DARPA-funded GALE project, a more nuanced metric called Human-Targeted Translation Edit Rate (HTER) is preferred. HTER measures the edits needed to transform an MT output into a post-edited version of itself, rather than comparing it to a pre-existing human reference. The challenge here is that HTER requires professional human editors, which brings back the cost and time constraints. This paper addresses the need for a scalable and cost-effective way to obtain data that can approximate HTER, specifically for ranking documents by quality.

Methods

The core innovation of this paper lies in leveraging Amazon Mechanical Turk (AMT) for data collection. Instead of hiring and training professional translators, the authors propose enlisting untrained AMT workers to perform the post-editing task. The process involves providing AMT workers with the original MT output and a human reference translation. Workers are instructed to modify the MT output to match the meaning and grammaticality of the human reference, using as few edits as possible.

A key challenge with crowdsourced data is quality control. To address this, the authors collect edits from multiple workers for each segment. They then explore different methods for combining these redundant edits, including taking the minimum, median, and average. Crucially, they introduce a "self-verification" approach where a small portion of each worker's edits are treated as a verification dataset. This allows them to evaluate how closely a worker's editing behavior matches that of a professional editor (the "gold-standard"). A worker's contribution is then weighted based on this self-verification, allowing for a more intelligent combination of edits to predict HTER.

Datasets

The study utilizes a dataset from the GALE project's Arabic-to-English MT task. This dataset consists of over 2,153 Arabic segments spread across 195 documents, categorized into four genres: broadcast conversations (BC), broadcast news (BN), newswire (NW), and blogs (WB). For evaluation, the authors also had access to "gold-standard" post-edits produced by LDC-hired professional translators.

Key Insights and Ideas

  • Crowdsourcing for HTER is feasible: The paper demonstrates that untrained workers on AMT can provide useful post-edits that approximate the behavior of professional editors, making HTER-like evaluations more accessible.
  • Quality control is paramount: The inherent variability in AMT worker quality necessitates robust methods for data aggregation and worker weighting.
  • Self-verification is effective: Using a small portion of a worker's own edits as a "gold-standard" to evaluate their overall performance is a practical and effective way to assign weights and improve prediction accuracy.
  • Document ranking is the target: The paper emphasizes that predicting the ranking of documents by quality is the primary goal, rather than precisely predicting the HTER score itself. This is a more achievable and practically useful objective for system development.

Key Results

The paper shows that using AMT-collected data, combined with their proposed weighted averaging method based on self-verification, significantly outperforms traditional automatic metrics like TER in predicting document rankings. Specifically, the "turk_wtd-average" method (weighted average using self-verification) consistently achieves higher rank correlation coefficients across all genres compared to the TER baseline. Even with a small verification subset (e.g., 10-20% of segments), the AMT-based methods achieve substantial improvements. The "turk_oracle" (hypothetical perfect worker) also performs very well, suggesting that identifying the best worker is a key to accurate HTER prediction.

Limitations and Future Work

The authors acknowledge that while their approach significantly outperforms automatic metrics, it's not perfect. They note that the "oracle" scenarios represent an unachievable upper bound. A limitation is that their current self-verification method primarily focuses on matching the professional editor's edit count, not necessarily the quality of the edits themselves.

Future work suggested includes explicitly identifying and filtering out undesirable worker behaviors, such as workers not editing the MT output at all or using the human reference instead of editing the MT output. They propose characterizing worker submissions by their distance to both the MT output and the human reference to build a more complete worker profile and further guard against poor data quality.

Summary updated

BibTeX
@inproceedings{zaidan2010predicting,
  author = {Omar F. Zaidan and Chris Callison-Burch},
  title = {{Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure