PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts

Chang Liu, Daniel Dahlmeier, Hwee Tou NgEMNLP 2010 · OpenReview

Abstract

We present PEM, the first fully automatic metric to evaluate the quality of paraphrases, and consequently, that of paraphrase generation systems. Our metric is based on three criteria: adequacy, fluency, and lexical dissimilarity. The key component in our metric is a robust and shallow semantic similarity measure based on pivot language N-grams that allows us to approximate adequacy independently of lexical similarity. Human evaluation shows that PEM achieves high correlation with human judgments.

Introduction

Paraphrase Generation (PG) is a critical task in Natural Language Processing, enabling systems to rewrite text while preserving its original meaning. However, evaluating PG systems has historically been challenging. Standard metrics like BLEU, designed for Machine Translation, are inadequate for paraphrasing because they reward lexical overlap. A perfect paraphrase should ideally use different words, yet BLEU would penalize such an output. Consequently, researchers have relied on costly and subjective manual evaluations. To address this, we introduce PEM (Paraphrase Evaluation Metric), a fully automatic metric that aligns closely with human intuition.

Methodology

A high-quality paraphrase must satisfy three fundamental criteria: adequacy (meaning preservation), fluency (grammatical correctness), and lexical dissimilarity (novelty in expression).

Semantic Similarity via Pivot Languages

The core innovation of PEM is the Bag of Pivot language N-grams (BPNG). To measure adequacy without relying on word-for-word similarity in the source language, we utilize a third "pivot" language. If two different English phrases translate to the same representation in a pivot language (e.g., Chinese or French), they are likely semantically equivalent. We employ Confusion Networks to represent these translations, providing a robust semantic bridge that handles vocabulary variations and translation uncertainty.

Feature Fusion

Fluency is assessed using a standard 4-gram language model, while lexical dissimilarity is calculated based on the lack of overlap between the original and paraphrased sentences. Because these features interact non-linearly—for instance, high dissimilarity is only valuable if adequacy is also high—we use a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel to integrate these components into a single quality score ranging from 1 to 5.

Evaluation

We evaluated PEM using the Multiple-Translation Chinese Corpus (MTC), which contains 993 Chinese news sentences with multiple English translations serving as paraphrases. The BPNG component was trained on the FBIS Chinese-English Corpus, a large parallel dataset of approximately 250,000 sentences. Fluency was modeled using the SRILM toolkit.

Results

PEM demonstrates a remarkably high correlation with human judgment, significantly outperforming traditional metrics. At the sentence level, PEM achieved a Pearson correlation of 0.8073.

Sentence-level correlation between PEM scores and human judgment.

When evaluating at the system level—ranking entire paraphrase generation models—the correlation with human intuition increased to 0.9867, establishing PEM as a highly reliable benchmarking tool.

System-level correlation between PEM scores and human judgment.

Conclusion

PEM represents a significant advancement in the automatic evaluation of paraphrases. By leveraging pivot languages to capture semantic similarity, it overcomes the limitations of lexical-based metrics like BLEU. While current limitations include handling complex sentence-level restructuring and a dependency on large parallel corpora, PEM provides a consistent and objective "gold standard" for the development of future paraphrase generation systems.

Summary updated

Abstract

We present PEM, the first fully automatic metric to evaluate the quality of paraphrases, and consequently, that of paraphrase generation systems. Our metric is based on three criteria: adequacy, fluency, and lexical dissimilarity. The key component in our metric is a robust and shallow semantic similarity measure based on pivot language N-grams that allows us to approximate adequacy independently of lexical similarity. Human evaluation shows that PEM achieves high correlation with human judgments.
BibTeX
@inproceedings{liu2010pem,
  author = {Chang Liu and Daniel Dahlmeier and Hwee Tou Ng},
  title = {{PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://openreview.net/forum?id=Hk4A4zzdWS},
}
Figure