Abstract
Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods. In this work, we are the first to incorporate information about predicate-argument structure of source sentences (namely, semantic-role representations) into neural machine translation. We use Graph Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders and achieve improvements in BLEU scores over the linguistic-agnostic and syntax-aware versions on the English--German language pair.
Introduction
Neural Machine Translation (NMT) systems are highly fluent but often struggle with preserving the exact underlying meaning of a sentence. A frequent and severe mistake is "argument switching"—for example, confusing who gave a present with who received it.
While previous research has successfully injected syntactic information (grammar and sentence trees) into NMT to help guide translations, semantic information has been largely ignored in the neural era. Semantic representations, such as PropBank-style semantic roles, abstract away from surface-level phrasing to map out "who did what to whom." This research hypothesizes that explicitly feeding this predicate-argument structure to an NMT system can act as a powerful linguistic bias, helping the model generalize better and avoid catastrophic meaning-altering errors.
Methodology
The authors build upon standard attention-based encoder-decoder architectures, experimenting with both Bidirectional RNN (BiRNN) and CNN encoders. To inject semantics, they use a pre-trained dependency-based semantic role labeler to parse the source English sentences. This creates a graph where words are nodes, and edges represent semantic roles (e.g., linking the predicate "gave" to the argument "John" with the role of "giver").
They then stack Graph Convolutional Networks (GCNs) on top of the base encoder. The GCN updates the representation of each word based on its semantic neighbors in the graph. Because semantic parsers can be noisy, the GCN uses learned scalar gates to weight the importance of each edge, allowing the network to down-weight potentially incorrect predictions. The study also explores stacking semantic GCNs on top of syntactic GCNs to combine both linguistic signals.
Experimental Setup
The models were evaluated on the WMT16 English-to-German dataset across two settings:
- Resource-poor setting: Using only the News Commentary v11 data (~226,000 sentence pairs).
- High-resource setting: Using the full WMT16 training set (~4.5 million sentence pairs).
- Evaluation:
newstest2015was used for validation, andnewstest2016served as the final test set.
Key Insights and Results
Performance Gains
- Small Dataset: Adding the semantic GCN improved the BiRNN baseline from 14.9 to 15.6 BLEU.
- Large Dataset: The semantic GCN improved the strong BiRNN baseline from 23.3 to 24.5 BLEU. Interestingly, the semantic bias provided a larger boost in the high-resource setting, suggesting that modeling semantic influence effectively requires substantial data.
- Complementary Signals: The best results were achieved by stacking semantic GCNs on top of syntactic GCNs, reaching 24.9 BLEU. This demonstrates that grammar and meaning provide distinct, mutually beneficial signals.
Qualitative Analysis
Manual analysis revealed that semantic GCNs significantly improved the model's ability to choose correct prepositions and maintain grammatical consistency in complex sentences. By recognizing semantic roles (e.g., identifying a "Buyer" vs. a "Direction"), the model avoids common translation pitfalls.
Conclusion
This work demonstrates that incorporating explicit semantic structure via GCNs enhances NMT performance and meaning preservation. While BLEU scores show clear improvements, future work will focus on more nuanced evaluation metrics and testing the approach on language pairs with more divergent syntactic structures, where semantic abstraction may prove even more critical.
Summary updated