A Dependency-Based Neural Reordering Model for Statistical Machine Translation

Christian Hadiwinoto, Hwee Tou Ng

Introduction

Word reordering remains a significant challenge in Statistical Machine Translation (SMT), particularly between languages with divergent syntactic structures like Chinese and English. Traditional phrase-based SMT systems often rely on local reordering models or sparse features that struggle to generalize across diverse linguistic contexts. This paper, presented at AAAI 2017, introduces a neural reordering model that leverages dependency-based syntactic information to improve translation quality.

The Dependency-Based Neural Reordering Model

The proposed approach integrates a neural network-based reordering model directly into the SMT decoding process. Unlike traditional models that use discrete, sparse features, this model employs continuous vector representations (embeddings) for words, Part-of-Speech (POS) tags, and dependency labels.

Neural architecture and dependency parse tree illustration.

The architecture consists of two specialized classifiers:

  1. Head-Child Classifier: Determines whether a child node should be swapped with its head node in the dependency tree.
  2. Sibling Classifier: Predicts the relative order of two nodes sharing the same parent.

These classifiers utilize dependency-driven embeddings, which are pre-trained on large corpora using syntactic context rather than simple linear adjacency. This allows the model to capture deeper grammatical relationships. During decoding, the neural network provides a reordering score that helps the SMT system select the most linguistically plausible word order.

Experimental Evaluation

The model was evaluated on standard Chinese-to-English translation tasks using the NIST MT evaluation sets (2002-2006, 2008). The training data included 8.8 million sentence pairs from the LDC, and embeddings were trained on the 2-billion-word Chinese Gigaword corpus.

The system achieved a statistically significant improvement of +0.57 BLEU points over a state-of-the-art baseline that already incorporated advanced dependency features. Notably, this hybrid SMT+Neural approach outperformed a standalone Neural Machine Translation (NMT) system trained on the same data by 0.58 BLEU points at the time of publication.

Key Insights and Qualitative Results

The transition from discrete to continuous features proved crucial. By representing syntactic roles as vectors, the model effectively handles rare grammatical structures that would otherwise result in "sparse feature" errors. Furthermore, the choice of initialization for word embeddings was found to be critical; dependency-context initialization significantly outperformed random initialization.

Comparison of translation outputs from different model configurations.

As shown in the translation examples, the inclusion of dependency-based neural reordering (DDP+NR) leads to more coherent and grammatically correct English sentences compared to models relying solely on standard reordering methods.

Conclusion and Future Work

This research demonstrates that explicit syntactic information remains highly valuable even in the era of neural networks. While the current implementation is tailored for SMT, the underlying principles of using dependency-based classifiers and embeddings are applicable to broader end-to-end neural architectures. Future work involves extending this framework to other language pairs and exploring more integrated ways to combine syntactic parsing with neural translation.

Summary updated

Abstract

In machine translation (MT) that involves translating between two languages with significant differences in word order, determining the correct word order of translated words is a major challenge. The dependency parse tree of a source sentence can help to determine the correct word order of the translated words. In this paper, we present a novel reordering approach utilizing a neural network and dependency-based embeddings to predict whether the translations of two source words linked by a dependency relation should remain in the same order or should be swapped in the translated sentence. Experiments on Chinese-to-English translation show that our approach yields a statistically significant improvement of 0.57 BLEU point on benchmark NIST test sets, compared to our prior state-of-the-art statistical MT system that uses sparse dependency-based reordering features.
BibTeX
@inproceedings{hadiwinoto2017dependency,
  author = {Christian Hadiwinoto and Hwee Tou Ng},
  title = {{A Dependency-Based Neural Reordering Model for Statistical Machine Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10499},
}
Figure