Abstract
Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods. In this work, we are the first to incorporate information about the predicate-argument structure of source sentences (namely, semantic-role representations) into neural machine translation. We use Graph Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders and achieve improvements in BLEU scores over the linguistic-agnostic and syntax-aware versions on the English–German language pair.
Introduction
Standard Neural Machine Translation (NMT) models are typically "linguistically agnostic," treating sentences as simple sequences of tokens. While effective at capturing surface patterns, these models often struggle with meaning preservation. A frequent and severe error is "argument switching," where the model confuses the roles of participants—for instance, translating "the dog chased the cat" as "the cat chased the dog."
While previous attempts to mitigate this have focused on syntactic structures, this research proposes that semantics (the underlying meaning) provides a more robust abstraction. Semantic representations can group disparate surface realizations of the same idea into a single structure, potentially helping the model generalize better across different ways of expressing the same concept.
Methodology: Semantic GCNs
The core innovation of this work is a hybrid architecture that integrates traditional sequence models with Graph Convolutional Networks (GCNs) to bake semantic structure directly into the sentence encoder.
The Encoding Pipeline
- Base Encoder: The model first processes the source sentence through a standard BiRNN or CNN to capture local word context.
- Semantic Labeling: Source sentences are annotated with semantic roles (PropBank-style), identifying predicates (verbs) and their arguments (e.g., Agent, Patient, Instrument).
- GCN Layer: The semantic roles are treated as a graph where words are nodes and semantic relationships are edges. The GCN allows each word to update its representation by communicating with its semantic neighbors. To handle the inherent noise in automatic labeling, the authors implement a gating mechanism that allows the model to down-weight unreliable edges. The GCN also accounts for the directionality of relationships (e.g., distinguishing the giver from the receiver).
- Decoder Integration: These semantically-enriched representations are passed to a standard attention-based decoder to generate the final translation.
Experimental Results
The models were evaluated on the English–German language pair using the News Commentary v11 and the large-scale WMT16 datasets.
Quantitative Performance
On the full WMT16 task, the semantic GCN model achieved 24.5 BLEU, a 1.2-point improvement over the BiRNN baseline (23.3). Notably, semantic information (+1.2 BLEU) outperformed syntactic information (+0.6 BLEU) on this dataset. The best results (24.9 BLEU) were achieved by stacking a semantic GCN on top of a syntactic one, suggesting that these two linguistic biases provide complementary value.
Qualitative Analysis
Qualitative analysis revealed that the model became significantly better at selecting correct prepositions by identifying the underlying semantic role of nouns. For example, it correctly distinguished when a noun was a "Buyer" (Role A2) versus a "Direction" (AM-DIR), leading to more accurate translations of the English "to" (e.g., using the German "an" instead of "nach").
Discussion and Future Work
The study demonstrates that semantic bias is particularly effective when scaled to large datasets, possibly because the model learns to truly exploit complex semantic graphs when provided with sufficient training data. However, the reliance on BLEU scores remains a limitation, as it may not fully capture the nuances of meaning preservation. Future work could involve human evaluations to better quantify these semantic gains and exploring the impact of this architecture on morphologically richer languages.
Summary updated