TL;DR
This paper introduces Meteor++, an enhanced machine translation evaluation metric that incorporates "copy knowledge" – words that are frequently preserved between a source and its paraphrased translation. By identifying and penalizing the omission or addition of these crucial "copy-words," Meteor++ demonstrates improved correlation with human judgments on standard MT evaluation benchmarks.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is a critical task, and automatic metrics aim to mimic human assessment. While metrics like BLEU and Meteor have been successful, they often struggle to fully capture the nuances of translation quality, particularly when dealing with paraphrasing. The authors observe that in paraphrasing, certain words tend to remain unchanged – this is termed "copy knowledge." Existing MT evaluation metrics don't explicitly account for this phenomenon. If a candidate translation omits words that are consistently present in human references (i.e., "copy-words"), it might be missing vital information. Conversely, adding extraneous "copy-words" can also degrade quality. This paper aims to bridge this gap by developing an MT evaluation metric that explicitly leverages this "copy knowledge."
Methods
The core of this work is the development of Meteor++, an extension of the existing Meteor metric. The process involves two main steps:
- Copy Knowledge Extraction: The authors propose a statistical method to identify "copy-words." This involves analyzing paraphrase corpora (like Quora question pairs and WMT datasets with high human scores) to find words that frequently appear in both the source and its paraphrased equivalent. They define "copy-words" as those exceeding certain thresholds for raw count and co-occurrence probability. Initially, they used a simple bag-of-words approach, but later experiments focused on named entities (identified using NLTK) as a proxy for "copy-words" due to data limitations.
- Incorporating Copy Knowledge into Meteor: Meteor++ modifies the precision (P) and recall (R) calculations within the Meteor framework. Specifically, it introduces penalties for two scenarios:
- Recall Penalty: If "copy-words" present in the reference are missing from the candidate translation.
- Precision Penalty: If "copy-words" present in the candidate are not found in the reference.
These penalties are implemented using new formulas for P and R that discount scores based on the presence or absence of these identified "copy-words." A hyper-parameter
Xis introduced to smooth these calculations.
Datasets
The authors utilize the following datasets:
- Quora Dataset: A large dataset of potential question duplicate pairs, used for initial copy knowledge extraction.
- WMT Datasets (WMT15, WMT17): These are standard benchmarks for machine translation evaluation. The authors select candidate translations with high human scores from these datasets to form paraphrase pairs for copy knowledge extraction. WMT16 evaluation sets were used as development sets for tuning parameters.
Key Insights and Ideas
- "Copy Knowledge" is a Stable Signal: The observation that certain words are consistently preserved across paraphrases is a valuable signal for translation quality.
- Named Entities as a Proxy: In the absence of extensive high-quality paraphrase pairs, named entities emerge as a significant and practical category of "copy-words" for MT evaluation.
- Penalizing Omissions and Additions: Directly penalizing the loss of important "copy-words" from the reference or the addition of extraneous ones in the candidate provides a more nuanced evaluation than traditional n-gram overlap.
- Recall is Crucial for MT: The paper reiterates Meteor's emphasis on recall, arguing that it's vital for ensuring a translation covers the source content, and "copy-words" are key to this coverage.
Key Results
The primary result is that Meteor++ demonstrates improved performance over the original Meteor metric.
- Higher Pearson Correlation: Meteor++ achieved higher average segment-level Pearson correlation with human judgments (Direct Assessment scores) on both WMT15 and WMT17 evaluation sets compared to Meteor.
- Parameter Tuning: Experiments showed a positive correlation between the hyper-parameter
Xand Named Entity (NE) density, leading to specific values ofXbeing chosen for WMT17 (X=14) and WMT15 (X=8).
Limitations and Future Work
- Copy Knowledge Extraction Method: The authors acknowledge that their statistical method for extracting "copy-words" is simple and has a long way to go compared to "real" copy knowledge. They suggest exploring alignment on large-scale parallel monolingual corpora to extract more universal copy knowledge.
- Data Limitations: The scarcity of high-scoring paraphrase pairs in WMT datasets limited their ability to extract a broader set of "copy-words" beyond named entities.
- Hyper-parameter Tuning: While
Xwas tuned, the authors plan to further optimize formulas by training on separate datasets as their copy knowledge extraction becomes more refined.
Summary updated