Re-evaluating Machine Translation Results with Paraphrase Support

Liang Zhou, Chin-Yew Lin, Eduard H. HovyEMNLP 2006 · OpenReview

Abstract: In this paper, we present ParaEval, an automatic evaluation framework that uses paraphrases to improve the quality of machine translation evaluations. Previous work has focused on fixed n-gram evaluation metrics coupled with lexical identity matching. ParaEval addresses three important issues: support for paraphrase/synonym matching, recall measurement, and correlation with human judgments. We show that ParaEval correlates significantly better than BLEU with human assessment in measurements for both fluency and adequacy.

Introduction

For years, the Machine Translation (MT) community has relied on BLEU (Bilingual Evaluation Understudy). While revolutionary, BLEU suffers from a major "lexical identity" problem: it only rewards a system if its output matches the human reference text word-for-word. If a human writes "the car is fast" and the machine produces "the automobile is quick," BLEU assigns a score of zero for those words, even though the meaning is identical. This creates a lack of paraphrase support and makes it difficult to define a meaningful recall score, forcing metrics to rely on precision and crude workarounds like brevity penalties.

Methodology

ParaEval addresses these limitations through a two-tier matching strategy supported by an automatically generated paraphrase library.

Paraphrase Acquisition

To avoid the cost of manual labor, the authors used an unsupervised method to find paraphrases by looking at a massive Chinese-English parallel corpus. They identified English phrases that translated to the same Chinese phrase—a technique known as "pivoting." For example, if "bombing" and "explosion" both translate to the same Chinese word, they are added to a paraphrase set.

Paraphrase acquisition via pivoting between English and Chinese parallel corpora.

Two-Tier Matching and Clipping

The system first tries to match phrases in the candidate translation to the reference using the paraphrase set. Any remaining words are then matched using standard unigram lexical identity. To prevent systems from "gaming" the metric by repeating good words, ParaEval extends BLEU’s "clipping" philosophy to paraphrase sets: if a reference contains one instance of "bombing," the system only gets credit for one instance of any word in that paraphrase set (like "explosion").

Experimental Setup

The framework was evaluated using the NIST 2003 Chinese MT evaluation set, which includes 100 source documents, translations from eight different MT systems, and four human reference translations. Human assessments of "fluency" and "adequacy" provided by NIST were used to calculate Spearman rank-order coefficients to determine how well the metric's ranking of systems matched human judgment.

Key Results

ParaEval consistently outperformed BLEU in correlating with human assessments:

  • Fluency: ParaEval achieved a correlation of 0.7575, compared to BLEU’s 0.6978.
  • Adequacy: ParaEval reached 0.6918, while BLEU trailed at 0.6108.
  • Recall Power: By using paraphrases to expand the "correct" space, the authors calculated a meaningful recall score that correlated very strongly (0.7373) with human adequacy.
ParaEval scores effectively distinguish between human and machine translation performance.

As shown in the scoring distribution, ParaEval clearly separates human translations from machine translations, a key requirement for a robust evaluation metric. It also proved more stable across different reference sets, effectively acting as a "synthetic" set of additional references.

Conclusion

ParaEval demonstrates that moving beyond rigid lexical identity to incorporate semantic equivalence significantly improves the reliability of automatic MT evaluation. While noise remains in short-phrase extraction, the use of a large, automatically generated paraphrase library provides a scalable way to align automatic metrics more closely with human perception of fluency and adequacy.

Summary updated

Abstract

In this paper, we present ParaEval, an automatic evaluation framework that uses paraphrases to improve the quality of machine translation evaluations. Previous work has focused on fixed n-gram evaluation metrics coupled with lexical identity matching. ParaEval addresses three important issues: support for paraphrase/synonym matching, recall measurement, and correlation with human judgments. We show that ParaEval correlates significantly better than BLEU with human assessment in measurements for both fluency and adequacy.
BibTeX
@inproceedings{zhou2006re,
  author = {Liang Zhou and Chin-Yew Lin and Eduard H. Hovy},
  title = {{Re-evaluating Machine Translation Results with Paraphrase Support}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=BkbQvMG_WH},
}
Figure