Abstract
We present PEM, the first fully automatic metric to evaluate the quality of paraphrases, and consequently, that of paraphrase generation systems. Our metric is based on three criteria: adequacy, fluency, and lexical dissimilarity. The key component in our metric is a robust and shallow semantic similarity measure based on pivot language N-grams that allows us to approximate adequacy independently of lexical similarity. Human evaluation shows that PEM achieves high correlation with human judgments.
Introduction
Paraphrase Generation (PG) is a critical task in Natural Language Processing, enabling systems to rewrite text while preserving its original meaning. However, evaluating PG systems has historically been challenging. Standard metrics like BLEU, designed for Machine Translation, are inadequate for paraphrasing because they reward lexical overlap. A perfect paraphrase should ideally use different words, yet BLEU would penalize such an output. Consequently, researchers have relied on costly and subjective manual evaluations. To address this, we introduce PEM (Paraphrase Evaluation Metric), a fully automatic metric that aligns closely with human intuition.
Methodology
A high-quality paraphrase must satisfy three fundamental criteria: adequacy (meaning preservation), fluency (grammatical correctness), and lexical dissimilarity (novelty in expression).
Semantic Similarity via Pivot Languages
The core innovation of PEM is the Bag of Pivot language N-grams (BPNG). To measure adequacy without relying on word-for-word similarity in the source language, we utilize a third "pivot" language. If two different English phrases translate to the same representation in a pivot language (e.g., Chinese or French), they are likely semantically equivalent. We employ Confusion Networks to represent these translations, providing a robust semantic bridge that handles vocabulary variations and translation uncertainty.
Feature Fusion
Fluency is assessed using a standard 4-gram language model, while lexical dissimilarity is calculated based on the lack of overlap between the original and paraphrased sentences. Because these features interact non-linearly—for instance, high dissimilarity is only valuable if adequacy is also high—we use a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel to integrate these components into a single quality score ranging from 1 to 5.
Evaluation
We evaluated PEM using the Multiple-Translation Chinese Corpus (MTC), which contains 993 Chinese news sentences with multiple English translations serving as paraphrases. The BPNG component was trained on the FBIS Chinese-English Corpus, a large parallel dataset of approximately 250,000 sentences. Fluency was modeled using the SRILM toolkit.
Results
PEM demonstrates a remarkably high correlation with human judgment, significantly outperforming traditional metrics. At the sentence level, PEM achieved a Pearson correlation of 0.8073.
When evaluating at the system level—ranking entire paraphrase generation models—the correlation with human intuition increased to 0.9867, establishing PEM as a highly reliable benchmarking tool.
Conclusion
PEM represents a significant advancement in the automatic evaluation of paraphrases. By leveraging pivot languages to capture semantic similarity, it overcomes the limitations of lexical-based metrics like BLEU. While current limitations include handling complex sentence-level restructuring and a dependency on large parallel corpora, PEM provides a consistent and objective "gold standard" for the development of future paraphrase generation systems.
Summary updated