Abstract
This paper presents a novel approach to improve reordering in phrase-based machine translation by using richer, syntactic representations of units of bilingual language models (BiLMs). Our method to include syntactic information is simple in implementation and requires minimal changes in the decoding algorithm. The approach is evaluated in a series of Arabic-English and Chinese-English translation experiments. The best models demonstrate significant improvements in BLEU and TER over the phrase-based baseline, as well as over the lexicalized BiLM by Niehues et al. (2011). Further improvements of up to 0.45 BLEU for Arabic-English and up to 0.59 BLEU for Chinese-English are obtained by combining our dependency BiLM with a lexicalized BiLM. An improvement of 0.98 BLEU is obtained for Chinese-English in the setting of an increased distortion limit.
Introduction: The Reordering Challenge
In Statistical Machine Translation (SMT), "reordering" (or distortion) refers to the process of moving words from the source-language order into the correct target-language order. This is a significant hurdle when translating between languages with divergent syntactic structures, such as Arabic (often VSO) and English (SVO).
Standard Phrase-Based SMT (PBSMT) systems typically address this using linear distortion penalties or lexicalized reordering models. However, linear models ignore context, and lexicalized models often suffer from data sparsity, failing to generalize to infrequent words. While tree-based models offer better structural handling, they are computationally intensive. This research proposes a "middle ground" by integrating source-side dependency syntax into efficient Bilingual Language Models (BiLMs).
Methodology: Syntactified BiLMs
The proposed approach builds on the BiLM framework, which treats translation as a sequence of "bilingual tokens"—tuples consisting of a target word and the source words aligned to it.
The core innovation is the "syntactification" of these tokens. Instead of using raw words or simple Part-of-Speech (POS) tags, the authors enrich each bilingual token with its source-side dependency context. They represent tokens using combinations of:
- Pos: The word’s own POS tag.
- Parent (→): The POS tag of the word's parent in the dependency tree.
- Grandparent (→→): The POS tag of the word's grandparent.
- Siblings (+sibl): The POS tags of the closest left and right siblings.
By labeling units with their "syntactic family," the model captures structural invariants. For instance, a "Noun modified by a Verb" should follow similar reordering patterns regardless of the specific lexical items. This information is precomputed and integrated as a feature in the SMT decoder's log-linear model, requiring no changes to the search algorithm.
Experimental Results
The approach was evaluated on Arabic-English (4.4M sentence pairs) and Chinese-English (2.1M sentence pairs) tasks using NIST benchmarks.
Key Findings:
- Syntax as a Stable Signal: Dependency relations provide a more stable signal for reordering than raw words. The inclusion of the parent's POS tag was found to be the most critical factor in improving reordering accuracy.
- Performance Gains: In Arabic-English, the best dependency model improved BLEU by 0.73 points over the PBSMT baseline. In Chinese-English, combining lexicalized and dependency BiLMs yielded a 1.29 BLEU gain.
- Robustness to Search Space: Dependency BiLMs excel when the "distortion limit" (the maximum distance a word can move) is increased. For Chinese-English, raising the limit from 5 to 10 resulted in a 0.98 BLEU improvement over the lexicalized BiLM, demonstrating superior discrimination in complex reordering scenarios.
Conclusion
By injecting structural intelligence from dependency grammar into the PBSMT framework, this method achieves significant translation quality improvements without the overhead of full tree-based models. Future work may explore target-side constraints and more complex notions of "syntactic cohesion" to further refine how groups of words are reordered together.
Summary updated