Feasibility of Human-in-the-loop Minimum Error Rate Training

Omar Zaidan, Chris Callison-BurchEMNLP 2009

Introduction

A persistent challenge in Machine Translation (MT) is the "evaluation gap"—the discrepancy between automatic metrics like BLEU and actual human judgment. While Minimum Error Rate Training (MERT) is the standard for tuning MT systems, it relies heavily on these automatic proxies because real-time human evaluation is prohibitively slow and expensive. Zaidan and Callison-Burch (2009) propose a novel solution: RYPT, a human-in-the-loop metric that leverages reusable sub-structure judgments to bridge this gap affordably.

The RYPT Metric

The core innovation of RYPT is its shift from sentence-level evaluation to constituent-level analysis. Instead of scoring a full hypothesis against a reference, RYPT utilizes the source sentence's parse tree. It evaluates whether specific syntactic constituents (e.g., noun phrases or verb phrases) are translated acceptably within a candidate translation.

Syntactic constituent labeling and alignment in the RYPT metric.

As shown in Figure 4, source phrases are aligned to target substrings. Human annotators provide binary "YES/NO" labels indicating if the translation of a specific constituent is acceptable. The final RYPT score for a sentence is the ratio of "YES" labels to the total number of nodes in the parse tree.

Methods: Reusability and Percolation

To make human evaluation feasible within the iterative MERT process, the authors employ two key strategies:

  1. Reusable Database: By labeling <source substring, target substring> pairs rather than full sentences, the system can store and reuse judgments. Since MT systems often generate redundant sub-phrases across different iterations, a growing database eventually allows for "automatic" scoring of new hypotheses.
  2. Label Percolation: The hierarchical nature of parse trees allows for efficient label propagation. If a parent node is marked "NO," its children are likely also unacceptable. Conversely, "YES" labels can often be percolated downward. This significantly reduces the manual labeling effort required, allowing the system to infer labels for over 70% of the nodes in the tree.

The authors utilized Amazon Mechanical Turk (AMT) for data collection, demonstrating that high-quality judgments could be obtained for a fraction of the cost of professional HTER (Human-targeted Translation Edit Rate) evaluations.

Key Results

The study, conducted on the WMT08 German-English News dataset, revealed that RYPT is a superior predictor of human preference compared to BLEU.

Comparison of human preferences between RYPT and BLEU across 250 sentences.

As summarized in Figure 6, RYPT-tuned systems were preferred by humans significantly more often than BLEU-tuned systems. In head-to-head comparisons, humans preferred the RYPT-selected translation in 48.0% of cases, compared to 35.2% for BLEU. Furthermore, the cost of collecting the necessary labels via crowdsourcing was remarkably low ($81.44), proving the economic feasibility of the approach.

Conclusion

The RYPT metric demonstrates that human-in-the-loop tuning is not only possible but also highly effective. By decomposing translations into reusable components and employing smart label percolation, researchers can move beyond the limitations of BLEU without incurring massive costs. Future work may involve moving toward probabilistic percolation models to further refine the accuracy of inferred labels.

Summary updated

BibTeX
@inproceedings{zaidan2009feasibility,
  author = {Omar Zaidan and Chris Callison-Burch},
  title = {{Feasibility of Human-in-the-loop Minimum Error Rate Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
}
Figure