Multi-Source Syntactic Neural Machine Translation

Anna Currey, Kenneth Heafield

Abstract

We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.

Introduction

Neural Machine Translation (NMT) has achieved significant success using Recurrent Neural Networks (RNNs) and attention mechanisms. However, these models often struggle to capture the underlying hierarchical structure of language, known as syntax. Previous attempts to integrate syntax involved "linearizing" parse trees into flat strings of labels and words. While helpful, this approach introduces two major challenges: increased sequence length, which complicates tracking long-distance dependencies, and brittleness, where the model fails if the external parser produces an error.

Methods

The proposed Multi-Source Syntactic NMT architecture addresses these issues by utilizing both the plain sentence and its grammatical structure simultaneously.

Dual Encoders and Hierarchical Attention

The model employs two independent RNN encoders. One encoder processes the standard "sequential" source sentence, while the second processes the "linearized" parse (containing words and grammatical tags). The authors evaluated two parse variations: Lexicalized (words + tags) and Unlexicalized (tags only).

To combine these inputs, a hierarchical attention mechanism is used. Separate attention layers are applied to each encoder, and a "master" attention layer merges the resulting context vectors. This allows the decoder to dynamically weight the importance of raw words versus syntactic structure at each decoding step.

Experimental Setup

The model was evaluated on the WMT17 English→German task using 5.9 million parallel sentence pairs. Preprocessing included Byte Pair Encoding (BPE) and parsing via Stanford CoreNLP.

Results

The multi-source approach yielded substantial performance gains over baseline models. The lexicalized multi-source model achieved a BLEU score of 26.5 on the 2016 test set, representing a 1.5-point improvement over the standard seq2seq baseline.

Main BLEU results on WMT17 English-German task

Analysis

Robustness and Sentence Length

A key advantage of the multi-source architecture is its "safety net" effect. By maintaining a dedicated encoder for raw text, the model remains robust even when parsed data is missing or corrupted. In stress tests where parsed input was withheld, the multi-source model maintained high performance (~23 BLEU), whereas the syntax-only parse2seq baseline collapsed to 0.6 BLEU.

Furthermore, the inclusion of syntactic structure significantly improves translation quality for long sentences. While standard NMT performance typically degrades as sentence length increases, the multi-source model's performance is much more stable, particularly for sentences exceeding 50 words.

BLEU scores by sentence length across different models

Conclusion

This work demonstrates that multi-source NMT effectively integrates syntactic information without the brittleness of single-source parsed models. While the current implementation uses RNNs and requires external parsing overhead, the principle of hierarchical attention over dual encoders provides a robust framework for structural awareness in translation. Future work may explore applying these concepts to Transformer architectures or incorporating synthetic data to further enhance performance.

Summary updated

Abstract

We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.
BibTeX
@inproceedings{currey2018multi,
  author = {Anna Currey and Kenneth Heafield},
  title = {{Multi-Source Syntactic Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=rk-0HMzuWS},
}
Figure