Abstract: In this paper, we present ParaEval, an automatic evaluation framework that uses paraphrases to improve the quality of machine translation evaluations. Previous work has focused on fixed n-gram evaluation metrics coupled with lexical identity matching. ParaEval addresses three important issues: support for paraphrase/synonym matching, recall measurement, and correlation with human judgments. We show that ParaEval correlates significantly better than BLEU with human assessment in measurements for both fluency and adequacy.
Introduction
For years, the Machine Translation (MT) community has relied on BLEU (Bilingual Evaluation Understudy). While revolutionary, BLEU suffers from a major "lexical identity" problem: it only rewards a system if its output matches the human reference text word-for-word. If a human writes "the car is fast" and the machine produces "the automobile is quick," BLEU assigns a score of zero for those words, even though the meaning is identical. This creates a lack of paraphrase support and makes it difficult to define a meaningful recall score, forcing metrics to rely on precision and crude workarounds like brevity penalties.
Methodology
ParaEval addresses these limitations through a two-tier matching strategy supported by an automatically generated paraphrase library.
Paraphrase Acquisition
To avoid the cost of manual labor, the authors used an unsupervised method to find paraphrases by looking at a massive Chinese-English parallel corpus. They identified English phrases that translated to the same Chinese phrase—a technique known as "pivoting." For example, if "bombing" and "explosion" both translate to the same Chinese word, they are added to a paraphrase set.
Two-Tier Matching and Clipping
The system first tries to match phrases in the candidate translation to the reference using the paraphrase set. Any remaining words are then matched using standard unigram lexical identity. To prevent systems from "gaming" the metric by repeating good words, ParaEval extends BLEU’s "clipping" philosophy to paraphrase sets: if a reference contains one instance of "bombing," the system only gets credit for one instance of any word in that paraphrase set (like "explosion").
Experimental Setup
The framework was evaluated using the NIST 2003 Chinese MT evaluation set, which includes 100 source documents, translations from eight different MT systems, and four human reference translations. Human assessments of "fluency" and "adequacy" provided by NIST were used to calculate Spearman rank-order coefficients to determine how well the metric's ranking of systems matched human judgment.
Key Results
ParaEval consistently outperformed BLEU in correlating with human assessments:
- Fluency: ParaEval achieved a correlation of 0.7575, compared to BLEU’s 0.6978.
- Adequacy: ParaEval reached 0.6918, while BLEU trailed at 0.6108.
- Recall Power: By using paraphrases to expand the "correct" space, the authors calculated a meaningful recall score that correlated very strongly (0.7373) with human adequacy.
As shown in the scoring distribution, ParaEval clearly separates human translations from machine translations, a key requirement for a robust evaluation metric. It also proved more stable across different reference sets, effectively acting as a "synthetic" set of additional references.
Conclusion
ParaEval demonstrates that moving beyond rigid lexical identity to incorporate semantic equivalence significantly improves the reliability of automatic MT evaluation. While noise remains in short-phrase extraction, the use of a large, automatically generated paraphrase library provides a scalable way to align automatic metrics more closely with human perception of fluency and adequacy.
Summary updated