Introduction
Word reordering remains a significant challenge in Statistical Machine Translation (SMT), particularly between languages with divergent syntactic structures like Chinese and English. Traditional phrase-based SMT systems often rely on local reordering models or sparse features that struggle to generalize across diverse linguistic contexts. This paper, presented at AAAI 2017, introduces a neural reordering model that leverages dependency-based syntactic information to improve translation quality.
The Dependency-Based Neural Reordering Model
The proposed approach integrates a neural network-based reordering model directly into the SMT decoding process. Unlike traditional models that use discrete, sparse features, this model employs continuous vector representations (embeddings) for words, Part-of-Speech (POS) tags, and dependency labels.
The architecture consists of two specialized classifiers:
- Head-Child Classifier: Determines whether a child node should be swapped with its head node in the dependency tree.
- Sibling Classifier: Predicts the relative order of two nodes sharing the same parent.
These classifiers utilize dependency-driven embeddings, which are pre-trained on large corpora using syntactic context rather than simple linear adjacency. This allows the model to capture deeper grammatical relationships. During decoding, the neural network provides a reordering score that helps the SMT system select the most linguistically plausible word order.
Experimental Evaluation
The model was evaluated on standard Chinese-to-English translation tasks using the NIST MT evaluation sets (2002-2006, 2008). The training data included 8.8 million sentence pairs from the LDC, and embeddings were trained on the 2-billion-word Chinese Gigaword corpus.
The system achieved a statistically significant improvement of +0.57 BLEU points over a state-of-the-art baseline that already incorporated advanced dependency features. Notably, this hybrid SMT+Neural approach outperformed a standalone Neural Machine Translation (NMT) system trained on the same data by 0.58 BLEU points at the time of publication.
Key Insights and Qualitative Results
The transition from discrete to continuous features proved crucial. By representing syntactic roles as vectors, the model effectively handles rare grammatical structures that would otherwise result in "sparse feature" errors. Furthermore, the choice of initialization for word embeddings was found to be critical; dependency-context initialization significantly outperformed random initialization.
As shown in the translation examples, the inclusion of dependency-based neural reordering (DDP+NR) leads to more coherent and grammatically correct English sentences compared to models relying solely on standard reordering methods.
Conclusion and Future Work
This research demonstrates that explicit syntactic information remains highly valuable even in the era of neural networks. While the current implementation is tailored for SMT, the underlying principles of using dependency-based classifiers and embeddings are applicable to broader end-to-end neural architectures. Future work involves extending this framework to other language pairs and exploring more integrated ways to combine syntactic parsing with neural translation.
Summary updated