TL;DR
This paper introduces ITER, a novel machine translation evaluation metric that improves upon the standard Translation Edit Rate (TER) by incorporating optimizable edit costs and stem matching. ITER demonstrates significantly higher correlation with human judgments, suggesting it's a more nuanced and accurate measure of translation quality.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is crucial for its development and deployment. While human evaluation is the gold standard, it's slow and expensive. Automatic metrics aim to bridge this gap, but many, like BLEU, have limitations in correlating well with human perception of translation quality. Translation Edit Rate (TER) is a widely used metric that measures the number of edits (insertions, deletions, substitutions, shifts) needed to transform a machine translation into a reference translation. However, TER treats all edit operations equally, which doesn't reflect the varying effort human translators exert for different types of corrections. This paper addresses the need for a more refined automatic metric that better captures human judgment by introducing flexible edit costs and other improvements.
Methods
The core innovation of ITER is the introduction of optimizable edit costs. Instead of assigning a uniform cost of 1 to all edit operations (insertion, deletion, substitution, shift), ITER allows these costs to be learned from data. This is motivated by the observation that human post-editors spend different amounts of time and effort on different types of edits.
Beyond optimizable costs, ITER introduces several other improvements:
- Stem Matching: When a hypothesis word and a reference word differ but share the same stem (e.g., "played" vs. "playing"), ITER allows for character-level edits instead of a full word substitution. This is particularly useful for morphologically rich languages.
- Improved Normalization: ITER refines the normalization process to ensure scores remain within a sensible range (0-1), addressing potential issues with TER where scores could exceed 100%. The normalization factor considers the number of hypothesis tokens, stemmed tokens, and the minimum edit cost.
- Character-Level Edits (with Stem Matching): For words that match at the stem level, character-level edits are considered, offering finer-grained correction possibilities.
Datasets
The authors trained and tuned their metric on the WMT15 datasets and evaluated it on the WMT16 datasets for both out-of-English and to-English translation directions. They also participated in the WMT 2018 Metrics Shared Task, submitting results for the "no hybrids" test set.
Key Insights and Ideas
- Edit costs matter: The most significant finding is that allowing edit costs to be optimized leads to a substantial improvement in correlation with human judgments. This highlights that not all errors are equal in the eyes of a human evaluator.
- Stem matching adds nuance: The inclusion of stem matching allows for more accurate evaluation of morphologically diverse languages, preventing overly harsh penalties for minor variations.
- ITER is a strong contender: The metric shows promising results, outperforming TER and competing with other top-performing metrics in shared tasks.
Key Results
The paper reports that ITER provides much higher correlation (ranging from 9.62% to 32.50% improvement) for every language pair and target language compared to TER on the WMT16 test set. Even without stem matching for en-ru, ITER showed significant improvement, indicating the impact of optimized edit costs. In the WMT16 segment level evaluation, ITER consistently outperformed TER across various language pairs. For the ru-en direction, ITER provided the best result among the evaluated metrics.
Limitations and Future Work
- Out-of-English limitations: For out-of-English translations, the authors had to skip stemming due to the unavailability of reliable stemmers for target languages. This means ITER was evaluated using only word-level operations, similar to TER, in these cases.
- Resource constraints: Due to resource constraints, the authors could not evaluate the "hybrids" test set in the WMT 2018 task, which involved artificially created system outputs.
- Future work: The authors suggest that ITER offers flexibility for users to choose their own edit operation costs, allowing customization for specific needs. The paper also implies that further exploration of character-level operations and more sophisticated normalization techniques could be beneficial.
Summary updated