Refining Source Representations with Relation Networks for Neural Machine Translation

Wen Zhang, Jiawei Hu, Yang Feng, Qun Liu

TL;DR

This paper introduces a "Relation Network" (RN) module into the standard RNN-based Neural Machine Translation (NMT) architecture to address two key limitations: RNNs' tendency to forget long-range dependencies and the encoder's focus on individual words rather than their relationships. By explicitly modeling pairwise word relationships, their RNMT system achieves significant improvements in translation quality, especially for longer sentences.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) models, particularly those using the popular RNN encoder-decoder architecture, have achieved remarkable success. However, they face two primary challenges. Firstly, Recurrent Neural Networks (RNNs), even with gating mechanisms like GRUs and LSTMs, struggle to retain information from distant parts of the input sequence, which can be crucial for accurate translation. Secondly, the standard encoder processes words in isolation, neglecting the inherent relationships (syntactic, semantic) between them. While some NMT approaches have implicitly captured certain syntactic phenomena, explicitly modeling word relationships has shown promise in improving translation quality. This paper aims to tackle both these issues by enhancing the source sentence representation.

Methods

The core innovation is the introduction of a Relation Network (RN) module, placed between the encoder and the attention layer of a standard NMT architecture. This RN module refines the source word representations before they are fed into the attention mechanism and decoder. The RN itself is composed of three sub-layers:

  1. CNN Layer: This layer uses convolutional filters to capture local context around each word, effectively aggregating information from neighboring words. This addresses the "forgetting" problem by providing a richer local context.
  2. Graph Propagation (GP) Layer: This layer constructs a fully connected graph where each word's representation (output from the CNN layer) is a node. Messages are propagated along the edges, allowing each node to receive information from all other nodes. This explicitly models pairwise relationships between words, regardless of their distance.
  3. MLP Layer: A Multi-Layer Perceptron is used to transform the aggregated information from the GP layer into a refined representation suitable for the attention mechanism and decoder.

Crucially, this approach integrates the RN without altering the main encoder-decoder architecture and does not require external syntactic knowledge. The paper also mentions the use of residual connections and dense concatenation for further refinement of the final output.

Datasets

The paper uses two Chinese-to-English translation datasets:

  • IWSLT: A smaller dataset (44K sentences) from the tourism and travel domain.
  • NIST: A larger dataset consisting of 1 million Chinese-English parallel sentence pairs.

Key Insights and Ideas

  • Explicitly modeling word relationships is beneficial: The paper strongly suggests that moving beyond word-level processing to consider pairwise word relationships significantly enhances translation quality.
  • Combining CNNs and Graph Propagation for context: The layered approach of using CNNs for local context and GP for global relational reasoning is an effective way to build rich source representations.
  • RNs can enhance existing NMT architectures: The proposed RN module can be seamlessly integrated into established RNN-based NMT models, offering a flexible way to improve performance.
  • Longer sentences benefit most: The intuition that RNNs forget long-range dependencies is validated, with the RNMT system showing a marked advantage on longer source sentences where this forgetting is more pronounced.

Key Results

The proposed RNMT system consistently outperformed baseline systems on both the IWSLT and NIST datasets.

  • On the IWSLT dataset, RNMT achieved a 4.1 BLEU point improvement over the NMT+ baseline.
  • On the NIST dataset, RNMT showed a 1.3 BLEU point improvement over NMT+.
  • The paper highlights that RNMT's superiority is particularly evident on longer sentences, where its BLEU scores significantly surpass those of NMT+ and Transformer models.
  • Qualitative analysis of word alignments and translation examples suggests that RNMT captures more source information and produces more reasonable attention distributions, leading to better translations.

Limitations and Future Work

  • Limitations: The paper acknowledges that while RNNs are good at modeling order, both RNNs and CNNs can struggle with long-distance dependencies. Their proposed method aims to alleviate this by integrating CNNs with bidirectional RNNs. The paper also notes that the Transformer model, while strong, might have different implementation details affecting direct comparison on smaller datasets.
  • Future Work: The paper implicitly suggests that further exploration of different RN configurations, kernel sizes, and the number of relation layers could lead to even greater improvements. The successful integration of RNs also opens avenues for applying similar relational reasoning techniques to other sequence-to-sequence tasks.

Summary updated

Abstract

Although neural machine translation (NMT) with the encoder-decoder framework has achieved great success in recent times, it still suffers from some drawbacks: RNNs tend to forget old information which is often useful and the encoder only operates through words without considering word relationship. To solve these problems, we introduce a relation networks (RN) into NMT to refine the encoding representations of the source. In our method, the RN first augments the representation of each source word with its neighbors and reasons all the possible pairwise relations between them. Then the source representations and all the relations are fed to the attention module and the decoder together, keeping the main encoder-decoder architecture unchanged. Experiments on two Chinese-to-English data sets in different scales both show that our method can outperform the competitive baselines significantly.
BibTeX
@inproceedings{zhang2018refining,
  author = {Wen Zhang and Jiawei Hu and Yang Feng and Qun Liu},
  title = {{Refining Source Representations with Relation Networks for Neural Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=rqLpnSNgnz9},
}

Loading related content...

Figure