Abstract
We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.
Introduction
While Neural Machine Translation (NMT) has significantly advanced the state of the art, standard Recurrent Neural Network (RNN) models often struggle to capture the underlying hierarchical structure of language without explicit supervision. Previous attempts to integrate syntax have often relied on linearized parses—transforming tree structures into flat strings of labels and words. However, this approach introduces two primary challenges: increased sequence length, which degrades RNN performance, and inference fragility, where the model becomes overly dependent on the quality of an external parser.
The Multi-Source Framework
To address these limitations, we propose a multi-source framework that treats syntactic information as a complementary input rather than a replacement for the raw sequence. The model architecture consists of:
- Dual Encoders: Two identical RNN encoders process the input in parallel. One encoder receives the standard "sequential" source sentence, while the other receives the "linearized parse" (which may be lexicalized, containing words, or unlexicalized, containing only tags).
- Hierarchical Attention: This mechanism manages the dual inputs. Each encoder has its own attention mechanism to track relevant input segments. A second-level hierarchical attention then merges these outputs into a single context vector for the decoder.
By decoupling the raw text from the syntactic structure, the model can leverage structural cues when available while maintaining a reliable fallback to the sequential text.
Experimental Results
We evaluated our approach on the WMT17 English→German (EN→DE) news translation task using 5.9 million parallel sentence pairs. The proposed multi-source models consistently outperformed both the standard seq2seq and the single-source parse2seq baselines.
As shown in the results, the lexicalized multi-source model achieved 21.9 BLEU on the WMT17 test set, a significant improvement over the 20.8 BLEU achieved by the standard baseline. Interestingly, lexicalized parses (including words) performed slightly better than unlexicalized ones, suggesting that the model benefits from seeing words within their structural context.
Analysis of Sentence Length and Robustness
A common issue with incorporating syntax is the "long sentence paradox": adding syntactic tags increases sequence length, which typically hurts RNN performance. However, our multi-source model reverses this trend.
For sentences exceeding 50 words, the multi-source model maintained a BLEU score of approximately 15, whereas the single-source parse2seq baseline plummeted below 10. This suggests that the hierarchical attention allows the model to use syntactic structure to maintain a "global" view of long sentences that plain text lacks.
Furthermore, the model demonstrated remarkable robustness to parser failure. In stress tests where the parsed input was replaced with a null value at test time, the multi-source model experienced only a minor performance dip. In contrast, the single-source parsed baseline completely failed, dropping to near zero BLEU.
Conclusion
By employing a multi-source architecture with hierarchical attention, we can effectively integrate syntactic information into NMT without the traditional drawbacks of sequence lengthening or parser dependency. This approach provides a structural "safety net" that improves translation quality, particularly for the most challenging long-form sentences.
Summary updated