Automatic Evaluation of Machine Translation Based on Rate of Accomplishment of Sub-Goals

Kiyotaka Uchimoto, Katsunori Kotani, Yujie Zhang, Hitoshi IsaharaNAACL 2007

TL;DR

This paper introduces a novel method for automatically evaluating machine translation (MT) quality by assessing whether specific linguistic "sub-goals" are met within a translation, rather than just relying on surface-level similarity to reference translations. This approach shows significantly higher correlation with human judgments of translation quality, particularly adequacy, and offers a more nuanced evaluation than existing metrics.

What Problem Does This Paper Address?

Evaluating the quality of machine translation (MT) output is a notoriously difficult problem. Traditional automatic evaluation metrics, like BLEU and NIST, primarily measure the n-gram overlap between an MT system's output and one or more human-written "reference" translations. While useful for ranking systems, these methods can be fooled by translations that are lexically similar but semantically or grammatically flawed, or vice-versa. For instance, a translation might receive a low score despite being perfectly understandable and accurate to a human, or a high score despite containing critical errors. This paper aims to bridge this gap by developing an evaluation method that better captures human perception of translation quality, focusing on whether essential linguistic conditions for a good translation are met.

Methods

The core of this paper's contribution lies in its proposed method for evaluating MT quality based on the "rate of accomplishment of sub-goals." The process involves several key steps:

  • Constructing a Test Set with Sub-goals: Instead of just parallel sentences, the authors create a test set where each sentence is associated with one or more "yes/no" questions. These questions represent linguistic conditions or "sub-goals" that a high-quality translation should satisfy (e.g., "Is 'appear to' translated into an auxiliary verb such as 'youda'?").
  • Developing an Automatic Answer Determination System: To avoid manual annotation for every translation, the authors develop a system that automatically determines the "yes" or "no" answer to these sub-goal questions. This is achieved by defining patterns (sequences of words or phrases) that indicate whether a sub-goal has been met. The system checks for the presence of these patterns in the translated sentence.
  • Combining Sub-goal Accomplishment with Similarity Measures: The final evaluation score is a combination of two components:
  • The rate of accomplishment of the sub-goals (weighted).
  • The similarity between the translated sentence and its reference translation(s) (using various standard metrics like BLEU, NIST, etc.). This combination aims to leverage the strengths of both local, rule-based checks (sub-goals) and global, similarity-based measures.

  • Automatic Sub-goal Generation: The paper also proposes a method for automatically generating these sub-goal questions. This involves identifying common skip n-grams (sequences of words with gaps) that appear in multiple reference translations for a given source sentence. The sub-goal then becomes checking if these common n-grams are present in the MT output.

Datasets

The primary dataset used and expanded upon in this work is the JEIDA test set. This dataset is characterized by having parallel sentences paired with yes/no questions designed to assess specific linguistic phenomena. The authors expanded this test set by manually adding more questions to translations that exhibited serious errors not covered by the original questions. They also used translations from five commercial MT systems for development and evaluation.

Key Insights and Ideas

  • Sub-goals as a Proxy for Human Judgment: The fundamental idea is that human translators implicitly satisfy a set of linguistic conditions (sub-goals) to produce a good translation. By explicitly defining and checking these sub-goals, an automatic evaluation can better mimic human judgment.
  • Limitations of Pure Similarity Metrics: The paper clearly demonstrates with an example (Table 1) how purely similarity-based metrics can misrank translations, highlighting the need for more nuanced evaluation.
  • Correlation with Adequacy is Key: The proposed method shows a particularly strong correlation with human judgments of "adequacy" (whether the translation conveys the meaning of the source), suggesting it captures semantic accuracy better than fluency alone.
  • Automatic Sub-goal Generation is Feasible: The development of an automatic method for generating sub-goal questions based on common phrases in reference translations is a significant step towards a fully automated evaluation system.
  • Complementarity of Local and Global Information: The proposed evaluation score, which combines sub-goal accomplishment (local linguistic checks) with similarity scores (global lexical overlap), is more effective than either component alone.

Key Results

  • The proposed method, "Our method (automatic)" and "Our method (full automatic)," consistently shows significantly higher correlation coefficients with human judgments of fluency and adequacy compared to conventional methods (WER, BLEU, NIST, and combinations thereof).
  • For example, with five reference translations, the correlation with adequacy reached 0.90 for "Our method (upper bound)" and 0.69 for "Our method (full automatic)," compared to 0.54* for the conventional combined method. The asterisk () indicates statistical significance.
  • The expansion of the JEIDA test set led to an increase in correlation coefficients, from 0.48 to 0.51 for fluency and from 0.63 to 0.66 for adequacy.
  • The accuracy of the automatic yes/no answer determination using patterns was high, reaching 97.6% on the development set and 82.8% on the evaluation set.

Limitations and Future Work

  • Pattern-Based Answer Determination: While effective, the automatic determination of answers relies on predefined patterns. The paper acknowledges that constructing these patterns can be difficult for certain types of questions, leading to some questions being skipped. The accuracy on the evaluation set (82.8%) also indicates room for improvement.
  • Complexity of Sub-goal Generation: The automatic sub-goal generation method focuses on common skip n-grams. Future work aims to improve this by considering sentence complexity and alignment information between source and translated sentences.
  • Expanding the Test Set: The authors plan to expand the test set to further improve the "upper bound" of their method.
  • Improving Automatic Estimation: Further research is needed to enhance the accuracy of automatic sub-goal generation and the estimation of sub-goal accomplishment rates.
  • Evaluating Against Normalized Data: The authors express interest in evaluating their method against cross-judge normalized data in the future, as human judgments can be noisy.

Summary updated

BibTeX
@inproceedings{uchimoto2007automatic,
  author = {Kiyotaka Uchimoto and Katsunori Kotani and Yujie Zhang and Hitoshi Isahara},
  title = {{Automatic Evaluation of Machine Translation Based on Rate of Accomplishment of Sub-Goals}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure