Unsupervised Quality Estimation Model for English to German Translation and Its Application in Extensive Supervised Evaluation

Aaron Li-Feng Han, Derek F. Wong, Lidia S. Chao, Liangye He, Yi LüThe Scientific World JOURNAL 2014 · DOI

TL;DR

This paper introduces a novel unsupervised machine translation (MT) evaluation model, nLEPOR, that estimates translation quality without relying on human-generated reference translations. By leveraging part-of-speech (POS) sequences and tunable parameters, nLEPOR aims to overcome the language-bias and cost issues of traditional metrics, demonstrating improved correlation with human judgments in both unsupervised and supervised settings.

What Problem Does This Paper Address?

Evaluating the quality of machine translation (MT) is crucial for tracking progress and improving systems. Traditional MT evaluation metrics, like BLEU, METEOR, and TER, primarily rely on comparing system-generated translations (hypotheses) against human-created reference translations. However, these reference-aware methods suffer from several drawbacks:

  • Cost and Availability: Creating high-quality reference translations is expensive and time-consuming, making them impractical for frequent evaluations or in scenarios with limited resources.
  • Language Bias: Metrics designed for one language pair may not perform well on others due to differences in linguistic structures and features.
  • Feature Engineering: Some metrics use too few linguistic features (leading to criticism from linguists), while others use too many, impacting repeatability and complexity.
  • Reference Sufficiency: It's challenging to determine how many reference translations are sufficient to avoid evaluation bias, as a single reference might not capture all acceptable translations.

This paper addresses these limitations by proposing an unsupervised MT evaluation metric that doesn't require reference translations and aims to be more robust across different language pairs.

Methods

The core of the proposed approach is the nLEPOR (n-gram based quality estimation metric for machine translation with augmented factors) model. It introduces several key innovations:

  • Unsupervised Evaluation: The primary goal is to evaluate translation quality using only the source sentence and the system's hypothesis translation, eliminating the need for reference translations.
  • Part-of-Speech (POS) Sequences: Instead of relying solely on surface words, nLEPOR utilizes POS tag sequences. The authors argue that POS tags capture syntactic and morphological information, providing a more robust linguistic feature. They also address the challenge of aligning POS tagsets between English and German by creating a complementary mapping and extending the universal POS tagset to 57 tags.
  • Augmented Factors: nLEPOR incorporates several factors beyond simple n-gram matching:
  • Enhanced Length Penalty (LP): This penalty accounts for both shorter and longer hypothesis sentences compared to the source sentence, unlike BLEU's brevity penalty which primarily penalizes shorter sentences.
  • N-gram Position Difference Penalty (NPosPenal): This factor penalizes differences in the positional alignment of matched POS n-grams between the hypothesis and source. The authors propose a POS alignment algorithm to find the best matches.
  • Harmonic Mean of n-gram Precision and Recall (H(αR, βP)): This component calculates precision and recall on a sentence level (not corpus-level like BLEU) and uses a harmonic mean with tunable weights (α for recall, β for precision) to balance these aspects.
  • Tunable Parameters: The weights for different n-gram orders (w_n) and the α/β weights for precision and recall are tunable. This allows the model to adapt to different language pairs and potentially mitigate language bias.
  • Enhanced Model (hLEPOR): A further enhancement, hLEPOR, combines scores from word-level evaluation and POS-level evaluation, allowing for a more comprehensive assessment. It also introduces tunable weights for these two components.

Datasets

The authors used corpora from the Workshop on Statistical Machine Translation (WMT) for their experiments:

  • WMT11 corpora: Used for training the models and tuning parameters.
  • WMT12 corpora: Used for testing the models in both unsupervised and supervised settings.
  • WMT13 corpora: Used to evaluate the enhanced hLEPOR model.

The paper also mentions the use of the Berkeley parser for German and the German Negra Treebank for POS tagset analysis.

Key Insights and Ideas

  • POS Sequences as a Linguistic Feature: The core insight is that POS sequences, when aligned and compared, can serve as a valuable linguistic feature for MT evaluation, offering a more robust signal than surface words alone, especially in an unsupervised setting.
  • Addressing Language Bias with Tunable Parameters: The ability to tune parameters (e.g., weights for n-grams, precision/recall balance, word/POS combination) is a key idea for making the evaluation metric more adaptable and less prone to language-specific biases.
  • Unsupervised Evaluation is Feasible and Competitive: The paper demonstrates that unsupervised MT evaluation can achieve correlation scores comparable to, and in some cases exceeding, traditional reference-aware metrics, especially when reference translations are scarce or costly.
  • Conciseness in Linguistic Features: The authors highlight the advantage of their method in using a concise set of linguistic features (surface words and POS sequences) compared to metrics that might rely on extensive external resources (like stemming or synonym dictionaries).

Key Results

  • Unsupervised Performance: In unsupervised evaluations on WMT11 and WMT12 corpora, nLEPOR (both nLEPORA and nLEPORB) consistently achieved higher Spearman rank correlation scores with human judgments compared to reference-aware metrics like BLEU, METEOR, and AMBER. For instance, on WMT11 training data, nLEPORB achieved 0.60, while BLEU was 0.44 and METEOR was 0.30.
  • Supervised Performance: When applied to traditional supervised evaluation tasks (using reference translations), nLEPORA also demonstrated strong performance, yielding the highest average correlation scores on several language pairs in WMT11 and WMT12. For example, on WMT11, it achieved an average score of 0.77 across eight language pairs.
  • Enhanced Model (hLEPOR): The hLEPOR model, which combines word and POS evaluation, further improved performance, achieving the highest correlation scores on WMT13 shared tasks for English-to-other language pairs, outperforming other metrics in both Spearman and Pearson correlation.
  • Language Pair Performance: The results suggest that English-to-other language pairs are more challenging for MT evaluation, but the proposed metrics, with their tunable parameters, show robustness across various language directions.

Limitations and Future Work

  • Language Pairs: While the paper explores English-to-German and several other European language pairs, the authors suggest future work to test the robustness of their algorithms on Asian languages like Chinese, Korean, and Japanese.
  • Multireference Evaluation: The current work primarily focuses on scenarios where reference translations are absent or limited. Future work could consider how their models perform with multiple reference translations.
  • Semantic Similarity: The paper acknowledges that evaluating MT from the aspect of semantic similarity is a complex area that warrants further exploration.
  • Computational Cost: While aiming for conciseness, the POS alignment and matching processes might still incur computational costs, though the paper doesn't explicitly detail these.

The authors have made their developed algorithms freely available for research purposes.

Summary updated

Abstract

With the rapid development of machine translation (MT), the MT evaluation becomes very important to timely tell us whether the MT system makes any progress. The conventional MT evaluation methods tend to calculate the similarity between hypothesis translations offered by automatic translation systems and reference translations offered by professional translators. There are several weaknesses in existing evaluation metrics. Firstly, the designed incomprehensive factors result in language-bias problem, which means they perform well on some special language pairs but weak on other language pairs. Secondly, they tend to use no linguistic features or too many linguistic features, of which no usage of linguistic feature draws a lot of criticism from the linguists and too many linguistic features make the model weak in repeatability. Thirdly, the employed reference translations are very expensive and sometimes not available in the practice. In this paper, the authors propose an unsupervised MT evaluation metric using universal part-of-speech tagset without relying on reference translations. The authors also explore the performances of the designed metric on traditional supervised evaluation tasks. Both the supervised and unsupervised experiments show that the designed methods yield higher correlation scores with human judgments.
BibTeX
@article{han2014unsupervised,
  author = {Aaron Li-Feng Han and Derek F. Wong and Lidia S. Chao and Liangye He and Yi Lü},
  title = {{Unsupervised Quality Estimation Model for English to German Translation and Its Application in Extensive Supervised Evaluation}},
  journal = {The Scientific World JOURNAL},
  year = {2014},
  doi = {10.1155/2014/760301},
}
Figure