Choosing the Right Translation: A Syntactically Informed Classification Approach

Simon Zwarts, Mark Dras

Introduction

In the field of Machine Translation (MT), the Multi-Engine Machine Translation (MEMT) paradigm seeks to improve output quality by selecting the best candidate from multiple translation systems. A common strategy involves comparing a standard phrase-based translation with a version generated through syntactic reordering—a process where the source sentence is rearranged to mirror the target language's structure. While reordering often improves fluency, it can occasionally introduce errors. This paper investigates a classification-based approach to determine when to accept a reordered translation and when to revert to the baseline.

Problem Statement

Standard evaluation metrics like BLEU are effective for document-level assessment against human references but are notoriously unreliable for sentence-level selection in the absence of a reference. The core challenge is identifying "syntactically abnormal" translations that should be rejected. The authors propose a binary classifier to decide between a standard phrase-based output and a syntactically reordered candidate.

Methodology

The selection task is framed as a binary classification problem using Support Vector Machines (SVM) and Maximum Entropy (MaxEnt) models. The researchers evaluated three categories of features:

  1. Internal Indicators: Metrics derived from the MT engine, such as language model probabilities and alignment scores.
  2. Source Language Indicators: Features analyzing the source sentence's complexity, including sentence length and parse tree probabilities (using the Collins and Link parsers).
  3. Target Language Indicators: Features analyzing the grammatical health of the output translation to detect structural abnormalities.

Dataset and Evaluation

The study utilized the Europarl corpus, focusing on translation pairs between English and Dutch. The training set comprised approximately 30,000 sentence pairs, with 9,000 reserved for testing. Ground truth for the classifier was established using a heuristic based on n-gram precision as a proxy for sentence-level BLEU.

Breakdown of sentence pairs by BLEU comparability and reordering performance.

As shown in the data breakdown, a significant portion of the dataset consists of "BLEU comparable" pairs where one system clearly outperformed the other, providing a robust basis for training the classifier.

Results and Discussion

The experimental results revealed several key insights into the efficacy of syntactic features in translation selection.

The Importance of the Target Side

One of the most significant findings was that source-side indicators—such as the difficulty of parsing the input sentence—had negligible predictive power. The complexity of the source does not necessarily correlate with the failure of reordering rules. Instead, the "Target Language Indicators" proved to be the most reliable judges. If a parser struggles to process the translated output, it serves as a strong signal of poor translation quality.

Synergy and Accuracy

While internal MT features alone provided only marginal improvements over a random baseline, the addition of target-side syntactic features significantly boosted performance.

Classification accuracy for English-Dutch and Dutch-English learners using SVM and Maximum Entropy.

In the Dutch-to-English task, the baseline system scored 0.208 BLEU. By using an SVM-based picker to choose between the original and reordered versions, the score rose to 0.238. This improvement demonstrates that syntactic features and internal likelihood scores are complementary; together, they form a more effective decision-making tool than either could alone.

Performance in the Extreme Range

Although the overall accuracy of the classifier was modest, it demonstrated exceptional reliability at the extremes. When the classifier reported high confidence in its decision, its accuracy in identifying the worst translations jumped to over 90%. This suggests that the model is particularly adept at catching "train wreck" translations, even if it still struggles to distinguish between two "good" candidates.

Conclusion and Future Work

This research demonstrates that syntactically informed classification can materially improve translation quality in MEMT systems. By using multiple parsers to judge the grammatical integrity of the output, the system can effectively filter out poor reordering choices.

Future work will address the computational overhead of running full syntactic parsers in real-time environments and explore how this binary selection model can be scaled to choose between a larger array of MT engines. Additionally, the authors suggest applying these syntactically informed techniques to more modern MT architectures beyond phrase-based systems.

Summary updated

Abstract

One style of Multi-Engine Machine Translation architecture involves choosing the best of a set of outputs from different systems. Choosing the best translation from an arbitrary set, even in the presence of human references, is a difficult problem; it may prove better to look at mechanisms for making such choices in more restricted contexts.
BibTeX
@inproceedings{zwarts2008choosing,
  author = {Simon Zwarts and Mark Dras},
  title = {{Choosing the Right Translation: A Syntactically Informed Classification Approach}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2008},
  url = {https://openreview.net/forum?id=bOA6F9u4mq},
}
Figure