Abstract
Neural Machine Translation (NMT) has achieved remarkable success by treating translation as a sequence-to-sequence task. However, standard linear encoders often struggle to capture long-distance dependencies inherent in complex linguistic structures. This paper introduces Source Syntax-aware Neural Machine Translation (SSNMT), a novel approach that enriches the NMT encoder with structural information from dependency trees. We propose two complementary structural sequences: Child Enriched Structure (CES) and Head Enriched Structure (HES), which traverse dependency trees to capture global syntactic context. Our model significantly outperforms standard NMT baselines on Chinese-English and English-Japanese translation tasks, with particularly substantial gains observed in the translation of long and complex sentences.
The Challenge of Long-Distance Dependencies
Standard Neural Machine Translation (NMT) models typically treat a sentence as a simple sequence of words. While Recurrent Neural Networks (RNNs) and attention mechanisms are powerful, they often struggle to maintain "long-distance dependencies." In a complex sentence, the subject and the main verb might be separated by ten or twenty words. A standard encoder, processing the sentence word-by-word from left to right, might "forget" the subject by the time it reaches the verb. This leads to translations that are grammatically fluent but factually incorrect—often misattributing actions or losing the core meaning of the source sentence.
Source Syntax-aware Encoder
To address these limitations, the authors propose a "Source Syntax-aware" encoder that supplements the standard bidirectional RNN (Bi-RNN) with two additional structural sequences derived from a dependency tree:
- Child Enriched Structure (CES): This captures context from a "head" word to its "children" using a pre-order traversal of the dependency tree. This ensures that when the model looks at a child node (like an adjective), it already has context from its parent (the noun it modifies).
- Head Enriched Structure (HES): This captures context from "children" to their "head" using a post-order traversal, allowing the model to summarize the information of an entire subtree before processing the parent node.
The "Shortcut" Effect
By following the dependency tree, the model creates "shortcuts" for information flow. In a linear sequence, two related words might be 15 steps apart; in a dependency tree, they might be direct neighbors. This drastically reduces the "vanishing gradient" problem for syntactically related words. While standard NMT is excellent at local context (neighboring words), CES and HES provide the "global" syntactic context necessary to understand the "skeleton" of the sentence.
Architecture and Fusion
For every word in the source sentence, the encoder now generates four hidden states: two from the standard Bi-RNN, one from the CES-RNN, and one from the HES-RNN. These four vectors are fused using a Multi-Layer Perceptron (MLP) to create a single, "syntax-aware" representation for each word. This representation is then passed to a standard attention-based decoder. This fusion ensures that the model benefits from both local sequential context and global structural context.
Experimental Setup
The model was evaluated on two major translation tasks:
- Chinese-to-English: Using LDC datasets (1M sentence pairs) for training, with NIST evaluation sets (2005, 2006, 2008, 2012) for testing.
- English-to-Japanese: Using the ASPEC corpus (top 1.5M sentence pairs), following the WAT 2016 evaluation procedure.
- Dependency Parsers: The authors used in-house arc-eager dependency parsers trained on the Penn Treebank and Chinese Treebank to generate the source-side trees.
Key Results
The integration of source syntax yielded significant performance improvements across both tasks:
- Overall Improvement: On the Chinese-English task, SSNMT achieved an average gain of 1.06 BLEU points over the standard RNNsearch baseline. In English-Japanese tasks, SSNMT outperformed the previous state-of-the-art tree-based model by 0.94 BLEU points.
- Superiority in Long Sentences: The most dramatic results appeared in long sentences (50+ words). In the English-Japanese task, the SSNMT model outperformed the baseline by a massive 4.21 BLEU points for these long, difficult sentences. This confirms that the primary value of syntax is helping the model stay "on track" during complex translations.
- Complementary Structures: Ablation studies showed that CES and HES are both necessary. CES helps the model understand how a head word influences its modifiers, while HES helps the model aggregate the meaning of complex phrases into a single head word.
Conclusion and Future Work
SSNMT demonstrates that explicitly modeling source-side syntax can significantly improve NMT performance, particularly for long-distance dependencies. However, the model's performance is currently tethered to the quality of the dependency parser. If the parser makes a mistake in the "skeleton" of the sentence, the NMT model receives flawed structural information. Future directions include incorporating more granular information from the dependency trees, such as arc-labels (e.g., identifying specifically which word is the "subject" vs. "object") and Part-of-Speech (POS) tags to further refine the encoder's understanding.
Summary updated