Refining Source Representations with Relation Networks for Neural Machine Translation

Wen Zhang, Jiawei Hu, Yang Feng, Qun Liu

Abstract: Neural Machine Translation (NMT) often struggles with long-range dependencies and the "forgetting" problem inherent in sequential architectures. This paper, presented at COLING 2018, proposes a Relation Network Layer (RNL) to refine source representations by explicitly modeling pairwise word relationships. By integrating local context via CNNs and global context via relational reasoning, the proposed RNMT model achieves significant improvements in translation quality and robustness to sentence length without requiring external syntactic parsers.

Introduction

Traditional Neural Machine Translation (NMT) systems typically rely on Recurrent Neural Networks (RNNs), such as LSTMs or GRUs, to encode source sentences. While effective, these architectures face two primary challenges: the forgetting problem and a lack of structural awareness. As sentence length increases, the hidden state's ability to retain information from the beginning of the sequence diminishes. Furthermore, the sequential nature of RNNs makes it difficult to capture non-local dependencies, such as the relationship between a subject and a verb separated by a long prepositional phrase.

While some approaches attempt to incorporate linguistic structure through external syntactic trees, these methods are limited by the availability and accuracy of parsers for various languages. This paper introduces a Relation Network Layer (RNL) that learns these dependencies end-to-end, providing a more robust and flexible representation for the attention mechanism.

The Relation Network Layer (RNL)

The proposed RNL acts as an intermediary between the encoder and the attention mechanism. It is designed to transform the initial hidden states into a richer set of features that encapsulate both local and global dependencies. The RNL consists of three primary sub-layers:

  1. CNN Layer: A convolutional layer captures local, phrase-level context by sliding a window over the word representations. This ensures that the subsequent relational reasoning is performed on context-aware "objects" rather than isolated words.
  2. Graph Propagation (GP) Layer: This layer serves as the core of the RNL. It treats the sentence as a fully connected graph where every word is an object. A Multi-Layer Perceptron (MLP) computes relationship scores between every possible pair of words. By aggregating these pairwise relations, the model updates each word's representation with global context.
  3. MLP Layer: A final transformation maps the relational features into a format optimized for the decoder's attention mechanism.
The architecture of the proposed Relation Network Layer (RNL), detailing the CNN, GP, and MLP sub-layers.

Experimental Evaluation

The model was evaluated on several benchmarks, including the NIST Chinese-English dataset (1.25M pairs) and WMT English-German datasets. The RNMT model consistently outperformed the standard RNNsearch baseline. On the NIST task, RNMT achieved an average BLEU score of 39.21, compared to 37.73 for the baseline.

Robustness to Sentence Length

A key advantage of the Relation Network is its ability to mitigate the forgetting problem. By providing direct "shortcuts" between all word pairs, the model maintains a strong signal even in very long sequences. Analysis shows that while baseline performance drops significantly for sentences exceeding 50 words, RNMT maintains high accuracy, demonstrating the effectiveness of explicit relational modeling.

Comparison of BLEU scores between the baseline RNNsearch* and the proposed RNMT across varying sentence lengths.

Comparison with Supervised Syntax

Interestingly, RNMT outperformed models that were explicitly provided with human-annotated grammar trees (e.g., BiRNN+GCN). This suggests that the task-specific relationships learned by the RNL are more effective for translation than generic linguistic rules. Additionally, the model showed a lower Alignment Error Rate (AER), indicating that its internal attention mechanism more accurately reflects real-world word alignments.

Conclusion

The Relation Network Layer provides a powerful mechanism for refining source representations in NMT. By combining local convolutional features with global pairwise relations, the model overcomes the limitations of sequential RNNs. While the $O(n^2)$ computational complexity of pairwise comparisons is a consideration, the gains in translation quality and robustness—especially for long sentences—make it a compelling architecture for complex translation tasks. Future work may explore the integration of these relational ideas more deeply into non-recurrent architectures like the Transformer.

Summary updated

BibTeX
@inproceedings{zhang2018refining,
  author = {Wen Zhang and Jiawei Hu and Yang Feng and Qun Liu},
  title = {{Refining Source Representations with Relation Networks for Neural Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=rqLpnSNgnz9},
}
Figure