WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation

Julian Chow, Lucia Specia, Pranava MadhyasthaACL 2019

TL;DR

This paper introduces WMDO, a novel metric for evaluating machine translation quality that extends the Word Mover's Distance (WMD) by incorporating word order. By penalizing unnatural word sequences, WMDO aims to better capture translation fluency and achieves competitive or superior performance compared to existing metrics.

What Problem Does This Paper Address?

Evaluating machine translation (MT) quality is a persistent challenge. Traditional metrics like BLEU rely heavily on exact string matching, which can be too rigid and fail to recognize semantically equivalent translations that differ in wording. While metrics like METEOR and character-based approaches offer more flexibility, they still often focus on surface-level similarities. More recent methods leverage word embeddings to capture semantic similarity, but many overlook the crucial aspect of word order, which significantly impacts translation fluency. The authors identify this gap: existing semantic-based metrics don't adequately account for the natural flow and grammatical structure of a translated sentence, leading to potentially misleading evaluations.

Methods

The core of this paper is the introduction of WMDO, an extension of the Word Mover's Distance (WMD).

  • Word Mover's Distance (WMD): WMD, a metric originally proposed for comparing documents, treats sentences as distributions of word embeddings. It calculates the minimum "cost" to transform one document's word distribution into another's, essentially measuring how much "work" is needed to move words from one sentence to match the other in semantic space. This is framed as an Earth Mover's Distance problem. A key characteristic of WMD is that it does not consider word order.
  • WMDO (WMD with word order): To address WMD's lack of word order awareness, WMDO introduces a "fragmentation penalty." This penalty is applied after the standard WMD score is calculated. The idea is to penalize translations where words appear in a different order than the reference translation. This penalty is inspired by METEOR's fragmentation penalty, aiming to avoid double-penalizing. The penalty is calculated based on the number of "chunks" (sequences of adjacent words that match in both the translation and reference) relative to the total number of unigrams in the translation. A higher number of chunks (indicating more fragmentation) leads to a larger penalty. The authors also introduce a weighting factor, $\delta$, to control the influence of this penalty.

Datasets

The experiments were conducted using the WMT17 segment-level into-English datasets. These datasets contain data from seven different source languages, with each text including a reference translation and a machine translation, along with human annotations for quality assessment.

Key Insights and Ideas

  • Semantic Space is Viable for MT Evaluation: The paper reinforces the idea that semantic spaces, represented by word embeddings, are a promising avenue for evaluating machine translation quality, moving beyond simple string matching.
  • Word Order is Crucial for Fluency: The most significant insight is that simply matching words semantically isn't enough. The order in which words appear is critical for a translation to sound natural and fluent, a factor that WMDO explicitly addresses.
  • WMD's Limitation: The paper clearly demonstrates that standard WMD, while good at capturing semantic similarity, can assign perfect scores to grammatically awkward or nonsensical translations if the words are semantically aligned (as shown in Figure 2).
  • Fragmentation Penalty Effectiveness: The proposed fragmentation penalty effectively penalizes out-of-order words, leading to a more nuanced evaluation that aligns better with human judgment.
  • Parameter Tuning Matters: The optimal value for the $\delta$ parameter (controlling the word order penalty) can vary across language pairs, suggesting that a one-size-fits-all approach might not be ideal for all scenarios.

Key Results

  • WMDO Outperforms Standard WMD: Across most language pairs, WMDO consistently achieved higher correlation scores with human judgments than the standard WMD metric.
  • Competitive with State-of-the-Art: WMDO performed on par with or better than many other state-of-the-art evaluation metrics on the WMT17 dataset. For instance, in Table 4, WMDO (with $\delta=0.2$) often shows bolded values, indicating it was the best performing non-trained metric for that language pair.
  • Word Embeddings Impact: The choice of word embeddings significantly impacts performance. Pre-trained FastText embeddings generally outperformed word2vec embeddings, especially when combined with their n-gram character-based OOV resolution strategy.
  • OOV Strategies: For handling Out-Of-Vocabulary (OOV) words, using a single "zero vector" for all OOVs with standard WMD was found to be a competitive strategy, though FastText's inherent OOV handling was superior.

Limitations and Future Work

  • Parameter Sensitivity: The authors acknowledge that the optimal value of $\delta$ can vary by language pair. They suggest that this parameter might need tuning for different datasets or language combinations.
  • Sentence Length Influence: The paper notes that the penalty might be overly harsh for very short sentences where word order deviations are more pronounced. Future work could explore adjustments to the penalty to account for sentence length.
  • Cross-Lingual Embeddings: The authors propose extending their work to use cross-lingual word embeddings. This would allow for direct evaluation from source text to machine translation within a shared semantic space, potentially eliminating the need for parallel data and human evaluation in between.
  • Open-Source Implementation: The authors plan to release an open-source implementation of WMDO.

Summary updated

BibTeX
@inproceedings{chow2019wmdo,
  author = {Julian Chow and Lucia Specia and Pranava Madhyastha},
  title = {{WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure