Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation

Karolina Owczarzak, Declan Groves, Josef van Genabith, Andy WayNAACL 2006 · OpenReview

Abstract

Standard automatic machine translation (MT) evaluation metrics like BLEU and NIST often penalize valid translations that deviate from a single reference text's specific vocabulary or syntax. This paper presents a resource-light method for automatically generating paraphrased reference sentences using only the evaluation bitext (source and reference) via word alignment tools. By creating multiple references "for free," this approach significantly improves the correlation between automatic evaluation scores and human judgment without requiring expensive additional human translations or external linguistic databases.

Introduction

Automatic MT evaluation metrics are essential for the rapid development of translation systems. However, their reliance on exact n-gram matching between machine output and human references creates a "rigidity" problem: a system may produce a semantically perfect translation that is penalized simply for using a synonym not present in the reference. While multiple human references can mitigate this, they are costly to obtain. This research explores a self-contained method to derive domain-specific paraphrases directly from the text being evaluated.

The Problem: The Rigidity of Exact Matching

Metrics like BLEU and NIST are industry standards due to their efficiency. Yet, if a system translates a sentence using "large" instead of the reference's "big," it receives no credit for that word. Previous attempts to solve this using external databases like WordNet often introduce out-of-domain synonyms that do not fit the specific context of the text. There is a need for high-quality, context-aware paraphrases that can be generated without heavy manual labor or bulky external dependencies.

Methodology

The authors propose treating the evaluation data—the source text and its single human reference—as a "miniature parallel corpus." The process involves four key steps:

  1. Alignment: Standard statistical MT alignment software (GIZA++) is run on this mini-corpus to map source phrases to their target equivalents.
  2. Equivalence Sets: If a source word (e.g., question) aligns to multiple English words across the document (e.g., question, matter, issue), these English words are grouped into a synonym "equivalence set."
  3. Filtering: Closed-class words (prepositions, pronouns) are filtered out as they rarely function as true swappable synonyms.
  4. Substitution: New reference sentences are generated by swapping words in the original reference with synonyms from their respective equivalence sets.

This method ensures that paraphrases are extracted directly from the relevant domain, capturing nuances that general-purpose thesauri might miss.

Experimental Setup

The method was tested using a 2,000-sentence French-to-English corpus from Europarl. Two MT systems were evaluated: Pharaoh (a statistical phrase-based system) and Logomedia (a commercial rule-based system). To validate the automatic scores, a subset of 100 sentences from each system was scored by human judges for accuracy and fluency. Additionally, a larger 700,000-sentence corpus was used to analyze how the scale of the bitext affects paraphrase quality.

Results and Analysis

The inclusion of automatically generated paraphrases led to a measurable increase in evaluation scores for both systems. As shown in Figure 1, the Pharaoh system's BLEU score increased from 0.2131 with a single reference to 0.2407 with multiple paraphrased references.

BLEU and NIST scores for Pharaoh (PH) and Logomedia (LM) systems using single vs. multiple references.

Beyond simply raising scores, the method improved the reliability of the metrics. The correlation between automatic scores and human judgments increased when using multiple references. For the Pharaoh system subset, the Pearson's correlation for smoothed BLEU rose from 0.396 to 0.404, as detailed in Figure 2.

Correlation between automatic metrics and human judgment for the Pharaoh (PH) system subset.

Interestingly, the results confirmed a known bias in n-gram metrics: the rule-based Logomedia system received lower BLEU/NIST scores than the statistical Pharaoh system, despite being rated higher by human judges. The paraphrasing method helped narrow this gap by rewarding valid lexical choices that the statistical system might not have prioritized.

Discussion: Context and Variation

A major advantage of this bitext-derived approach is its sensitivity to context. In the Europarl dataset, the word "area" often aligned with "aspect." While a general thesaurus might suggest "land" or "territory," those would be incorrect in a political debate context. Furthermore, by aligning phrases, the system captures syntactic variations (e.g., "we would expect" vs. "we certainly expect") that word-level synonym databases cannot.

Experiments regarding corpus size revealed a trade-off: while a larger 700,000-sentence corpus improved the syntactic accuracy of the single best paraphrase (70% vs 59%), the smaller 2,000-sentence evaluation bitext actually provided better average accuracy across the entire set of generated paraphrases.

Conclusion and Future Work

This research demonstrates that high-quality, domain-specific paraphrases can be generated "for free" from the evaluation bitext itself. This method effectively improves the correlation between automatic metrics and human judgment. Future work will focus on using lattice structures to handle multiple simultaneous substitutions and applying these techniques to newer metrics like CDER and TER, which are inherently better at handling syntactic shifts.

Summary updated

BibTeX
@inproceedings{owczarzak2006contextual,
  author = {Karolina Owczarzak and Declan Groves and Josef van Genabith and Andy Way},
  title = {{Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
  url = {https://openreview.net/forum?id=OAZMSBKb2S},
}
Figure