Microsoft Research Treelet Translation System: NAACL 2006 Europarl Evaluation

Arul Menezes, Kristina Toutanova, Chris QuirkNAACL 2006 · OpenReview

Abstract: Arul Menezes, Kristina Toutanova, Chris Quirk. Proceedings on the Workshop on Statistical Machine Translation.

2006.

Introduction

In the mid-2000s, Statistical Machine Translation (SMT) was primarily driven by phrase-based models. These models treated language as a sequence of flat strings, which worked well for local context but struggled with global reordering—the structural shifts required when moving between languages with different syntactic rules (e.g., English to Spanish or German). To address these limitations, the Microsoft Research (MSR) team proposed a syntax-informed approach using dependency trees.

The Treelet Translation Architecture

The MSR system moves beyond contiguous phrases to "treelets"—connected subgraphs of a source dependency tree. By using syntax as a backbone, the system can capture grammatical relationships, such as the link between a head verb and its modifiers, regardless of their absolute position in a sentence.

Dependency tree alignment and treelet extraction for English-to-Spanish translation.

The extraction process involves parsing the source sentence and aligning it with the target language. The system identifies Minimal Translation Units (MTUs), which are the smallest sets of words that can be extracted without violating alignment constraints. This allows the model to handle non-compositional idioms and complex mappings without relying on excessively long, sparse phrases.

Log-Linear Reordering and Bilingual Models

A major contribution of this work is the transition to a log-linear reordering model. Previous iterations used decision trees, but the log-linear approach allows for a more flexible integration of features, including Part-of-Speech (POS) tags, lexical items, and relative head positions. This model predicts where a word should be placed relative to its "head" in the dependency structure.

Furthermore, the system utilizes bilingual n-gram models. Unlike standard language models that only look at the target text, these models simultaneously consider the sequence of MTUs in the source string, the target string, and the source dependency tree.

Evaluation and Results

The system was evaluated on the Europarl English-to-Spanish corpus for the NAACL 2006 workshop. A notable constraint of this evaluation was the exclusive use of provided bilingual data, eschewing the large monolingual corpora often used to inflate language model performance.

BLEU score improvements across system iterations on the development set.

As shown in the results, the introduction of bilingual MTU models and the log-linear reordering model provided steady improvements in BLEU scores. Specifically, the log-linear model increased the per-decision accuracy of word placement from 69% to 79%.

The "BLEU Gap" and Human Evaluation

Perhaps the most significant finding was the discrepancy between automated metrics and human judgment. While the system's BLEU scores were average compared to competitors, human evaluators ranked the MSR system 1st for in-domain data and 2nd for out-of-domain data. This "BLEU Gap" suggests that syntax-based systems produce more grammatically coherent and readable translations that n-gram-based metrics like BLEU fail to fully capture.

Additionally, the system demonstrated superior robustness. When moving to out-of-domain data, the MSR system's BLEU score dropped by only 3.61%, compared to an average drop of 5.11% across other competing systems.

Conclusion

The MSR Treelet Translation System demonstrates that incorporating syntactic structure into SMT provides significant advantages in reordering and robustness. While automated metrics may not always reflect these gains, human evaluation confirms the value of moving beyond flat string models toward dependency-aware translation.

Summary updated

BibTeX
@inproceedings{menezes2006microsoft,
  author = {Arul Menezes and Kristina Toutanova and Chris Quirk},
  title = {{Microsoft Research Treelet Translation System: NAACL 2006 Europarl Evaluation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
  url = {https://openreview.net/forum?id=DlkRlyxBDq},
}
Figure