Stochastic Iterative Alignment for Machine Translation Evaluation

Ding Liu, Daniel GildeaACL 2006

Abstract

Evaluating machine translation (MT) quality automatically remains a significant challenge due to the inherent variability of natural language. Traditional metrics like BLEU often fail to capture sentence-level fluency and require exact matches, limiting their correlation with human judgment. This paper introduces Stochastic Iterative Alignment (SIA), a metric that combines flexible sequence alignment with a language-independent stochastic word-matching technique. By penalizing gaps between aligned words and iteratively matching against multiple references, SIA achieves superior performance in assessing both adequacy and fluency, outperforming established metrics across various evaluation benchmarks.

Introduction

The gold standard for machine translation evaluation is human assessment, which is unfortunately slow, expensive, and difficult to scale. Consequently, the field has relied on automatic metrics such as BLEU, which measures n-gram overlap between machine outputs and human references. However, BLEU’s rigidity—requiring exact string matches and failing to account for word order variations—often results in poor correlation with human scores at the individual sentence level. While metrics like METEOR and ROUGE introduced "loose" matching and linguistic tools like stemmers, they often depend on language-specific resources like WordNet, which are not universally available.

Methodology

SIA addresses these limitations through three primary innovations designed to bridge the gap between adequacy (meaning) and fluency (naturalness):

  1. Modified String Alignment (Gap-based Scoring): Unlike simple subsequence matching, SIA utilizes a dynamic programming approach that incorporates a penalty for gaps between aligned words. This ensures that words appearing far apart in the translation compared to the reference are scored lower, effectively capturing the grammatical "flow" or fluency of the sentence.
  2. Stochastic Word Matching: To move beyond exact string matching without relying on hand-curated databases, SIA employs a "soft" matching approach. Word similarity is calculated based on probabilities derived from large bilingual corpora (e.g., using IBM Model 4). This statistical approach allows the metric to recognize synonyms or related terms in a language-independent manner.
  3. Iterative Alignment Scheme: Most metrics compare an MT output against multiple references and select only the single best match. SIA adopts a "greedy" iterative approach: it aligns the output against one reference, removes the matched words, and then attempts to align the remaining words against subsequent references. This ensures that every valid word in the MT output is credited, regardless of which reference it originated from.
Visual representation of word alignment between reference and machine translation outputs illustrating the flexibility of the matching scheme.

Experimental Setup

The performance of SIA was validated using data from the ACL05 MT Evaluation Workshop. The test set comprised seven sets of MT outputs (919 English sentences each) translated from Chinese. Human baselines were established by manually ranking each sentence on a 1-5 scale for both fluency and adequacy. For stochastic word matching, the model was trained on an English-French parallel corpus of 700,000 sentence pairs to establish word similarity probabilities.

Results and Discussion

The effectiveness of SIA was measured using Pearson’s correlation coefficients against human judgments. As shown in the results table below, SIA (denoted as 'S') consistently outperformed existing metrics.

Comparison of correlation coefficients for BLEU-3 (B-3), ROUGE (R_1, R_2), METEOR (M), and SIA (S) across Fluency (F), Adequacy (A), and Overall (O) categories.

At the sentence level, SIA achieved an overall correlation of 0.292, surpassing BLEU-3 (0.265) and METEOR (0.280). Notably, SIA demonstrated a significant advantage in evaluating Fluency, scoring 0.202 compared to the 0.167 achieved by both BLEU and METEOR. At the system level, SIA maintained its lead with a correlation of 0.835, compared to BLEU's 0.794.

Key insights from the study suggest that rewarding "tighter" sequences with fewer gaps mimics human perception of grammatical flow. Furthermore, the transition from rule-based linguistic tools to stochastic similarity enhances the metric's portability across different language pairs.

Conclusion

SIA represents a significant step forward in automatic MT evaluation by prioritizing fluency and flexibility. While the metric is more computationally intensive than n-gram methods and requires a high-quality bilingual corpus for training, its modular components—such as the iterative scheme—could potentially be integrated into other metrics. Future work may focus on optimizing the computational efficiency of the alignment process and exploring its application in low-resource language settings.

Summary updated

BibTeX
@inproceedings{liu2006stochastic,
  author = {Ding Liu and Daniel Gildea},
  title = {{Stochastic Iterative Alignment for Machine Translation Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
}
Figure