Refining Source Representations with Relation Networks for Neural Machine Translation

Wen Zhang, Jiawei Hu, Yang Feng, Qun Liu

Abstract

Neural machine translation (NMT) using the encoder-decoder framework has seen significant success but continues to struggle with "forgetting" distant information—a limitation of recurrent neural networks (RNNs)—and failing to model relationships between source words during encoding. This paper introduces Relation Networks (RN) to learn more robust source representations. By associating source words with one another, the Relation Network Layer (RNL) enhances the memorization capabilities of the RNN and preserves word relationships. These refined representations are then utilized by the attention mechanism during decoding. Experimental results demonstrate that this approach significantly improves translation performance over conventional models and even outperforms methods that incorporate supervised syntactic knowledge.

Introduction

Standard NMT models typically rely on RNNs to encode source sentences. However, RNNs face two primary challenges: the forgetting problem, where information from the start of a sentence is lost as the sequence grows, and the sequential bottleneck, which prevents the model from explicitly capturing non-sequential relationships (e.g., long-distance dependencies). While some researchers address this by incorporating external syntactic knowledge, such tools are not universally available across all languages. This work proposes a self-contained Relation Network Layer to overcome these hurdles.

Methods

The core contribution is the Relation Network Layer (RNL), positioned between the encoder and the attention mechanism. The RNL refines the encoder's hidden states through three distinct stages:

  1. CNN Layer: A convolutional layer captures local n-gram context, ensuring the model considers word groupings.
  2. Graph Propagation (GP) Layer: This layer treats the sentence as a fully connected graph. It calculates relations between every pair of words, allowing information to flow directly across the entire sequence regardless of distance.
  3. MLP Layer: A multi-layer perceptron refines these relational features for the attention component.
Architecture of the Relation Network Layer (RNL).

By utilizing residual connections, the RNL produces a rich, "refined" representation of the source text that captures both local and global dependencies.

Experimental Setup

The model (RNMT) was evaluated on several benchmarks:

  • NIST (Chinese-English): 1.25 million training pairs, tested on MT03-MT06.
  • WMT17 (English-German): A large-scale dataset with 5.6 million pairs.
  • WMT16 (English-German): Used for comparison against state-of-the-art syntactic models.

Key Results

The RNMT model demonstrated significant improvements in translation quality:

  • BLEU Score Gains: On the NIST task, the model achieved an average BLEU score increase of 1.48 points. On WMT17, it saw a gain of 1.7 points.
  • Robustness to Sentence Length: One of the most notable findings was the model's stability as sentence length increased. While standard RNN-based models often see a sharp decline in performance for long sentences, RNMT maintained high BLEU scores even for sequences exceeding 50 words.
BLEU score performance across different source sentence lengths.
  • Superiority over Syntactic Models: RNMT outperformed models like BiRNN+GCN, which rely on supervised syntactic knowledge. This suggests that the relationships learned automatically by the RNL are highly effective for translation tasks.
  • Improved Alignment: The Alignment Error Rate (AER) decreased by 1.1 points, indicating that the refined representations help the attention mechanism focus more accurately on relevant source words.

Conclusion and Future Work

The Relation Network Layer provides a powerful, data-driven way to model word relationships and mitigate the forgetting problem in NMT. While the $O(n^2)$ complexity of all-to-all relations is a consideration for very long sequences, the performance gains are substantial. Future research may explore integrating RNL with the Transformer architecture or developing more efficient graph pruning techniques to focus only on the most critical word relationships.

Summary updated

Abstract

Although neural machine translation with the encoder-decoder framework has achieved great success recently, it still suffers drawbacks of forgetting distant information, which is an inherent disadvantage of recurrent neural network structure, and disregarding relationship between source words during encoding step. Whereas in practice, the former information and relationship are often useful in current step. We target on solving these problems and thus introduce relation networks to learn better representations of the source. The relation networks are able to facilitate memorization capability of recurrent neural network via associating source words with each other, this would also help retain their relationships. Then the source representations and all the relations are fed into the attention component together while decoding, with the main encoder-decoder framework unchanged. Experiments on several datasets show that our method can improve the translation performance significantly over the conventional encoder-decoder model and even outperform the approach involving supervised syntactic knowledge.
BibTeX
@inproceedings{zhang2018refining,
  author = {Wen Zhang and Jiawei Hu and Yang Feng and Qun Liu},
  title = {{Refining Source Representations with Relation Networks for Neural Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=rqLpnSNgnz9},
}

Loading related content...

Figure