TL;DR
This paper introduces new features for machine translation (MT) evaluation that leverage source-language information and a novel training method called Maximum Correlation Training (MCT). By combining these new features, the authors achieve significantly better correlation with human judgments compared to existing metrics, offering a more nuanced evaluation of MT quality.
What Problem Does This Paper Address?
Evaluating machine translation (MT) systems is notoriously difficult. While metrics like BLEU are useful for large test sets, they often fail to accurately assess the quality of individual sentence translations. This is because a single source sentence can have many valid translations, and existing metrics primarily focus on surface-level similarities between the MT output and its reference translations, neglecting deeper semantic or structural aspects. The paper aims to improve sentence-level MT evaluation by developing more informative features and a better method for combining them.
Methods
The paper proposes three new types of features for MT evaluation:
- Source-sentence constrained n-gram precision: This feature refines traditional n-gram precision by only counting overlapping n-grams if their constituent words are aligned to the same words in the source sentence. This helps to ensure that n-gram matches reflect actual translation correspondences rather than coincidental overlaps.
- Source-sentence reordering metrics: These metrics assess how well the word order in the MT hypothesis and its references matches the word order in the source sentence. Two specific metrics are introduced: Pairwise Reordering Similarity (PRS) and Monotonic Reordering Ratio (MPR).
- Discriminative Unigram Precision based on POS (DUPP): This approach breaks down unigram precision by part-of-speech (POS) tags. The idea is that different POS categories (e.g., nouns, verbs) might have varying importance for translation adequacy and fluency, allowing for more flexible weighting.
The core innovation in combining these features is Maximum Correlation Training (MCT). Unlike previous methods that might indirectly optimize for correlation (e.g., by minimizing classification error), MCT directly optimizes the Pearson correlation between the combined metric's score and human judgments. This is achieved by treating the combined metric as a linear combination of individual features and using gradient ascent to find the weights that maximize this correlation.
Datasets
The experiments were conducted using data from the MT evaluation workshop at ACL05. This dataset includes:
- Seven sets of MT outputs (each with 919 English sentences translated from Chinese).
- Four references for each MT output.
- Two sets of human scores (fluency and adequacy) for each MT output.
The paper also mentions using a French-English parallel corpus for training stochastic word mapping and a Chinese-English newswire text corpus for training GIZA++.
Key Insights and Ideas
- Source sentence is valuable information: The paper strongly advocates for incorporating source-language information into MT evaluation metrics, demonstrating that it significantly improves correlation with human judgments.
- Alignment is key: Explicitly aligning source words to words in the MT hypothesis and references is crucial for developing more meaningful features, especially for n-gram precision and reordering.
- Direct optimization matters: Maximum Correlation Training (MCT), which directly optimizes the correlation with human judgments, proves to be more effective than classification-based approaches (like SVMs) for combining MT evaluation metrics.
- Diversity of features is beneficial: When combining metrics using MCT, features that capture different aspects of translation quality (e.g., n-gram precision, reordering, POS-specific unigrams) contribute more effectively than multiple metrics from the same category.
Key Results
- The proposed source-sentence constrained n-gram metrics (SSCN) and their stochastic variants (pSSCN) significantly outperform BLEU, especially in adequacy and overall evaluation.
- The best individual metric, pSSCN_u(2) (source-sentence constrained bigrams with stochastic word mapping and unioned constraints), achieved the best adequacy and overall scores and the second-best fluency score among all tested individual metrics.
- The combined metric using MCT achieved significantly better results than the best individual metrics. For adequacy and overall scores, the improvement was at the 99.5% confidence level, and for fluency, it was at the 96.9% confidence level.
- MCT, when used to combine discriminative unigram precisions based on POS (DUPP), demonstrated its ability to create fluency- or adequacy-oriented metrics by training on specific human judgment types.
- MCT outperformed a classification-based framework (CSVM) in combining metrics, even when CSVM had access to more human references for training.
Limitations and Future Work
- POS Tagger Robustness: The paper acknowledges potential worries about the robustness of POS taggers on noisy MT outputs but argues that POS tagging is generally more accurate than parsing and that accumulated POS counts mitigate the impact of individual errors.
- Convexity of Objective Function: The objective function for MCT is not always convex, meaning gradient ascent might converge to a local optimum. The authors address this by running gradient ascent from multiple random starting points, which yielded very similar results.
- Future Work: The paper suggests exploring higher-order n-grams within POS categories, though notes this could lead to sparsity issues. The success of MCT suggests further exploration of this direct optimization approach for MT evaluation. The authors also highlight the benefit of combining metrics with diverse properties, implying that future work could focus on identifying and developing such complementary features.
Summary updated