Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation

Yinuo Guo, Junfeng HuACL 2019

TL;DR

This paper introduces Meteor++ 2.0, an enhanced machine translation evaluation metric that incorporates syntactic-level paraphrase knowledge. By leveraging paraphrase databases, Meteor++ 2.0 captures meaning-preserving transformations beyond simple word matching, leading to improved correlation with human judgments.

What Problem Does This Paper Address?

Evaluating the quality of machine translation (MT) is crucial for tracking progress and comparing systems. While automatic metrics like BLEU and Meteor have been developed, they often struggle to fully capture the nuances of human language. The standard Meteor metric, for instance, primarily relies on lexical-level matching, including consecutive n-grams and synonyms. However, it overlooks syntactic transformations that can preserve meaning even when word order or function words change. This limitation can lead to lower scores for perfectly acceptable translations that deviate slightly from the reference in structure. The paper aims to bridge this gap by incorporating syntactic paraphrase knowledge into the Meteor framework.

Methods

The core innovation of Meteor++ 2.0 lies in its integration of syntactic-level paraphrase knowledge. The authors leverage the large Paraphrase Database (PPDB) to extract a subset of syntactic paraphrases, specifically focusing on "skip n-grams." These are phrase pairs where the components are not necessarily consecutive in the text, often involving placeholder words (represented by '_') that can match any word.

The process involves:

  1. Syntactic Paraphrase Table Acquisition: Extracting skip n-gram paraphrases from PPDB that contain at least one content word and have a limited length difference between the two phrases. This results in a table of approximately 27 million pairs.
  2. Integration into Meteor: A new "Syntax stage" is added to the existing Meteor matching process. This stage allows for matching phrases based on the extracted syntactic paraphrase table.
  3. Modified Beam Search: To handle the increased number of potential matches introduced by syntactic paraphrases without significantly degrading efficiency, a modified beam search strategy is employed. Syntactic matches are considered later in the search, and their validity is checked by looking backward to ensure all components of the paraphrase can be matched.
  4. Weighting: Different match types (Exact, Stem, Synonym, Paraphrase, Syntax) are assigned weights. For instance, the Syntax stage is given a weight of 0.4, while Exact matching retains a weight of 1.0.

Datasets

The experiments were conducted on evaluation datasets from the WMT15 and WMT17 Machine Translation Shared Tasks, specifically for to-English language pairs. The paper also mentions using the Paraphrase Database (PPDB) for extracting syntactic paraphrase knowledge.

Key Insights and Ideas

  • Syntactic Paraphrases Matter: The paper strongly suggests that syntactic variations, even those involving function words or word order changes, are crucial for accurate MT evaluation and are often overlooked by purely lexical metrics.
  • Skip N-grams as a Rich Source: Focusing on skip n-grams from PPDB proves effective in capturing a wide range of meaning-preserving syntactic transformations.
  • Balancing Precision and Recall: The introduction of syntactic knowledge aims to improve both precision and recall by allowing more flexible and semantically equivalent matches, ultimately leading to better correlation with human judgments.
  • Efficiency Considerations: The modified beam search highlights the practical challenge of incorporating richer linguistic knowledge into evaluation metrics and the need for efficient algorithms.

Key Results

Meteor++ 2.0, particularly when combined with copy knowledge (as in Guo et al., 2018), demonstrated improved segment-level Pearson correlation with human judgments on WMT15 and WMT17 evaluation datasets across most to-English language pairs. For example, on WMT17, Meteor++ 2.0 (copy + syntax) achieved scores like 0.722 for fi-en and 0.626 for ru-en, outperforming the baseline Meteor and Meteor++ (copy) metrics. The paper states that their proposed metric "outperforms all previous versions of Meteor" and "outperforms all prior Meteor-based metrics on almost every WMT15 and WMT17 to-English language pairs."

Limitations and Future Work

The authors acknowledge two main limitations:

  • Knowledge Quality: Despite filtering, some "unusual, inaccurate or highly context-dependent paraphrases" might still exist in the extracted syntactic paraphrase table. They suggest that their metric could potentially serve as a quality estimator for paraphrase tables in the future.
  • Mismatches due to Structure: Because syntactic-level knowledge focuses on sentence structure, mismatches can still occur. They propose that incorporating syntactic tools like parsing might help in better utilizing this knowledge.

Future work could involve exploring more sophisticated ways to leverage syntactic information, potentially through parsing, and investigating methods to further refine the quality of the syntactic paraphrase knowledge.

Summary updated

BibTeX
@inproceedings{guo2019meteor,
  author = {Yinuo Guo and Junfeng Hu},
  title = {{Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure