Precision and Recall of Machine Translation

I. Dan Melamed, Ryan Green, Joseph P. TurianNAACL 2003

TL;DR

This paper argues that traditional precision and recall metrics, adapted for machine translation (MT), offer a more intuitive and human-aligned evaluation than the popular BLEU score. By defining a "maximum matching size" that rewards contiguous word sequences (runs), their method provides insights into MT system weaknesses and demonstrates superior correlation with human judgments.

What Problem Does This Paper Address?

Evaluating the quality of machine translation (MT) is crucial for developing better systems. While human evaluation is the gold standard, it's time-consuming and expensive. This has led to the development of automatic evaluation metrics. Early metrics focused on simple word overlap, but more recent ones like BLEU (introduced by Papineni et al.) consider n-grams (sequences of words) to capture some sense of word order. However, BLEU scores can be opaque; a score of 0.317 doesn't readily tell you why a translation is good or bad. The authors aim to bridge this gap by proposing an evaluation framework based on the well-understood concepts of precision and recall, which they argue are more interpretable and better reflect human judgments of translation quality.

Methods

The core of the paper is adapting precision and recall to MT. The challenge lies in defining the "intersection" of a candidate translation and a reference translation. They visualize this as a bitext grid where each cell represents a word alignment.

  • Unigram-Based Measures: Initially, they consider a simple "maximum matching" approach from graph theory. This involves finding the largest set of word alignments (hits in the grid) such that no two alignments share the same row or column. This prevents double-counting words. Precision and recall are then calculated as the ratio of this maximum matching size to the length of the candidate or reference text, respectively.
  • Rewards for Longer Matches: To better capture word order, they introduce a novel method that rewards contiguous sequences of matching words (called "runs"). Each run is treated as an "aligned block," and its weight is the square of its length. The "maximum matching size" is then redefined to sum the weights of these aligned blocks. This "weighted maximum matching size" is used to compute precision and recall. This approach implicitly favors longer, correctly ordered sequences.
  • Multiple References: To account for the fact that a good translation can be expressed in many ways, they extend their method to handle multiple reference translations. This involves concatenating references and then computing the maximum matching, with a constraint that runs cannot span across different reference texts. The matching is then capped by the candidate length and the mean reference length.

Datasets

The authors used a corpus of six English translations of 728 Arabic sentences. Two of these were machine-generated ("candidate") translations, and four were human-generated ("reference") translations. They created "pseudo-documents" by concatenating 1 to 25 candidate sentences to test the evaluation metrics' reliability across different document lengths.

Key Insights and Ideas

  • Intuition Matters: The paper strongly advocates for evaluation metrics that are intuitively understandable, arguing that this aids in diagnosing MT system errors and guiding improvements.
  • Precision and Recall are Powerful: By adapting standard precision and recall, the authors demonstrate a framework that is not only statistically robust but also conceptually clearer than n-gram based metrics like BLEU.
  • Word Order is Key, but How? The novel "weighted maximum matching" approach effectively penalizes out-of-order words by giving disproportionately more weight to longer, contiguous matches. This is a more nuanced way to handle word order than simply counting n-grams.
  • Multiple References Improve Reliability: As expected, using multiple reference translations significantly boosts the reliability of their proposed metrics, as it accounts for linguistic variation.

Key Results

  • Their proposed precision and recall measures, based on weighted maximum matching, showed significantly higher correlation with human judgments (Adequacy and Fluency) compared to BLEU.
  • In the single-reference setting, their F-measure (the harmonic mean of precision and recall) was more than twice as reliable as BLEU.
  • Even with just one reference, their F-measure was more reliable than BLEU with three references for very short documents.
  • Precision and recall themselves showed lower correlations than the F-measure but were still better than BLEU.

Limitations and Future Work

  • Computational Cost: While they mention using a "randomized approximation for efficiency" in rare cases, the exact computation of maximum matching can be computationally intensive.
  • Generalizability: The paper focuses on MT, but they suggest their techniques could be applied to other NLP tasks involving text generation and summarization.
  • Visualization Tools: The authors mention they are developing tools for visualizing maximum matchings, which would further aid in understanding the metrics and MT system behavior.

Summary updated

BibTeX
@inproceedings{melamed2003precision,
  author = {I. Dan Melamed and Ryan Green and Joseph P. Turian},
  title = {{Precision and Recall of Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure