Information-Propogation-Enhanced Neural Machine Translation by Relation Model

Wen Zhang, Jiawei Hu, Yang Feng, Qun Liu

TL;DR

This paper introduces a "Relation Network" layer to enhance Neural Machine Translation (NMT) by explicitly modeling relationships between words in the source sentence. This approach improves the flow of information to the decoder, leading to more accurate translations, especially for longer sentences, outperforming existing state-of-the-art models.

What Problem Does This Paper Address?

Traditional Neural Machine Translation (NMT) models, particularly those based on Recurrent Neural Networks (RNNs) like LSTMs or GRUs, struggle to capture long-range dependencies within a sentence. While attention mechanisms help, they primarily rely on sequential relationships. Convolutional Neural Networks (CNNs) have been introduced to speed up NMT and capture local features, but they don't fully address the issue of understanding complex, non-local relationships between words. This paper aims to bridge this gap by enabling NMT models to better understand and propagate information about these relational dependencies across the entire source sentence, leading to more contextually aware translations.

Methods

The core innovation is the introduction of a "Relation Network" (RN) layer integrated into the standard attention-based NMT encoder-decoder framework. This RN layer sits between the encoder's output and the attention mechanism. It comprises three components:

  • Convolutional Encoder (CENC): This component applies 1D CNNs to the initial word representations from the NMT encoder. This step aims to create richer, localized feature representations.
  • Graph Propagation Layer (GPL): This layer explicitly models relationships between pairs of source words. It concatenates the convolutional representations of word pairs and uses this to propagate information, effectively learning pairwise interactions.
  • Multi-Layer Perceptron Decoder (MLPDEC): This MLP layer processes the outputs from the GPL to generate an "entangled sequential representation" for each source word. This representation is designed to capture the relational information from all other words in the sentence.

This "entangled sequential representation" is then fed into the attention layer of the NMT decoder, augmenting the standard recurrent sequential representation. The authors also explore using a Convolution Gated Recurrent Unit (CGRU) as an alternative to the 1D CNN encoder, combining convolutional operations within the GRU structure.

Datasets

The experiments were conducted on two Chinese-English translation datasets:

  • IWSLT Data: A smaller dataset consisting of 44K sentences from the tourism and travel domain.
  • NIST Data: A larger dataset comprising 1M Chinese-English parallel sentence pairs.

The NIST data was further broken down into several test sets (NIST 2002-2006) for evaluation.

Key Insights and Ideas

  • Explicit Relational Modeling is Crucial: The paper strongly suggests that simply relying on sequential processing (RNNs) or local feature extraction (CNNs) is insufficient for capturing the full semantic richness of a sentence. Explicitly modeling pairwise word relationships is key.
  • Information Propagation Enhances Context: The proposed Relation Network layer acts as an information highway, ensuring that relevant contextual information from distant words in the source sentence can effectively reach the decoder.
  • Robustness to Sentence Length: The relational approach demonstrates a significant improvement in handling longer sentences, where traditional models tend to lose information. This suggests that understanding sentence structure beyond simple sequential order is vital for long-range translation.
  • Improved Word Alignment: The relational model leads to more sensible and grammatically correct word alignments, indicating a deeper understanding of sentence structure and meaning.

Key Results

  • Significant BLEU Score Improvements: The proposed model, RNNSEARCH+RN, consistently outperformed baseline models, including the strong RNNSEARCH and even Google's Transformer model, on both the IWSLT and NIST datasets.
  • On the IWSLT dataset, the best configuration achieved a 4.05-point improvement over MOSES and 4.02 points over RNNSEARCH*.
  • On the 1M NIST training data, the model achieved a 1.13-point improvement over the strong NMT baseline RNNSEARCH* on average across four test sets.
  • Qualitative Improvements: Sample translations showed that the proposed model was better at capturing all information from the source sentence, avoiding under-translation and correctly interpreting complex relationships (e.g., adversative clauses).
  • Better Performance on Long Sentences: Figure 4 illustrates that while performance drops for all models with increasing sentence length, the proposed model shows a much more graceful degradation and achieves the best results for sentences longer than 50 words.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the addition of a relation layer, especially with convolutional and graph propagation components, likely increases computational complexity and training time compared to standard RNN-based NMT.
  • Future Work: The authors suggest exploring more complex relation network architectures and potentially integrating external linguistic knowledge (like dependency trees, as explored by others) to further enhance the model. They also mention the potential for using CGRU as an alternative to standard CNNs for the initial encoding.

Summary updated

Abstract

Even though sequence-to-sequence neural machine translation (NMT) model have achieved state-of-art performance in the recent fewer years, but it is widely concerned that the recurrent neural network (RNN) units are very hard to capture the long-distance state information, which means RNN can hardly find the feature with long term dependency as the sequence becomes longer. Similarly, convolutional neural network (CNN) is introduced into NMT for speeding recently, however, CNN focus on capturing the local feature of the sequence; To relieve this issue, we incorporate a relation network into the standard encoder-decoder framework to enhance information-propogation in neural network, ensuring that the information of the source sentence can flow into the decoder adequately. Experiments show that proposed framework outperforms the statistical MT model and the state-of-art NMT model significantly on two data sets with different scales.
BibTeX
@misc{zhang2017information,
  author = {Wen Zhang and Jiawei Hu and Yang Feng and Qun Liu},
  title = {{Information-Propogation-Enhanced Neural Machine Translation by Relation Model}},
  year = {2017},
  eprint = {1709.01766},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.01766},
}
Figure