Abstract
We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.
Introduction
Neural Machine Translation (NMT) has achieved significant success using Recurrent Neural Networks (RNNs) and attention mechanisms. However, these models often struggle to capture the underlying hierarchical structure of language, known as syntax. Previous attempts to integrate syntax involved "linearizing" parse trees into flat strings of labels and words. While helpful, this approach introduces two major challenges: increased sequence length, which complicates tracking long-distance dependencies, and brittleness, where the model fails if the external parser produces an error.
Methods
The proposed Multi-Source Syntactic NMT architecture addresses these issues by utilizing both the plain sentence and its grammatical structure simultaneously.
Dual Encoders and Hierarchical Attention
The model employs two independent RNN encoders. One encoder processes the standard "sequential" source sentence, while the second processes the "linearized" parse (containing words and grammatical tags). The authors evaluated two parse variations: Lexicalized (words + tags) and Unlexicalized (tags only).
To combine these inputs, a hierarchical attention mechanism is used. Separate attention layers are applied to each encoder, and a "master" attention layer merges the resulting context vectors. This allows the decoder to dynamically weight the importance of raw words versus syntactic structure at each decoding step.
Experimental Setup
The model was evaluated on the WMT17 English→German task using 5.9 million parallel sentence pairs. Preprocessing included Byte Pair Encoding (BPE) and parsing via Stanford CoreNLP.
Results
The multi-source approach yielded substantial performance gains over baseline models. The lexicalized multi-source model achieved a BLEU score of 26.5 on the 2016 test set, representing a 1.5-point improvement over the standard seq2seq baseline.
Analysis
Robustness and Sentence Length
A key advantage of the multi-source architecture is its "safety net" effect. By maintaining a dedicated encoder for raw text, the model remains robust even when parsed data is missing or corrupted. In stress tests where parsed input was withheld, the multi-source model maintained high performance (~23 BLEU), whereas the syntax-only parse2seq baseline collapsed to 0.6 BLEU.
Furthermore, the inclusion of syntactic structure significantly improves translation quality for long sentences. While standard NMT performance typically degrades as sentence length increases, the multi-source model's performance is much more stable, particularly for sentences exceeding 50 words.
Conclusion
This work demonstrates that multi-source NMT effectively integrates syntactic information without the brittleness of single-source parsed models. While the current implementation uses RNNs and requires external parsing overhead, the principle of hierarchical attention over dual encoders provides a robust framework for structural awareness in translation. Future work may explore applying these concepts to Transformer architectures or incorporating synthetic data to further enhance performance.
Summary updated