Abstract
Neural machine translation (NMT) using the encoder-decoder framework has seen significant success but continues to struggle with "forgetting" distant information—a limitation of recurrent neural networks (RNNs)—and failing to model relationships between source words during encoding. This paper introduces Relation Networks (RN) to learn more robust source representations. By associating source words with one another, the Relation Network Layer (RNL) enhances the memorization capabilities of the RNN and preserves word relationships. These refined representations are then utilized by the attention mechanism during decoding. Experimental results demonstrate that this approach significantly improves translation performance over conventional models and even outperforms methods that incorporate supervised syntactic knowledge.
Introduction
Standard NMT models typically rely on RNNs to encode source sentences. However, RNNs face two primary challenges: the forgetting problem, where information from the start of a sentence is lost as the sequence grows, and the sequential bottleneck, which prevents the model from explicitly capturing non-sequential relationships (e.g., long-distance dependencies). While some researchers address this by incorporating external syntactic knowledge, such tools are not universally available across all languages. This work proposes a self-contained Relation Network Layer to overcome these hurdles.
Methods
The core contribution is the Relation Network Layer (RNL), positioned between the encoder and the attention mechanism. The RNL refines the encoder's hidden states through three distinct stages:
- CNN Layer: A convolutional layer captures local n-gram context, ensuring the model considers word groupings.
- Graph Propagation (GP) Layer: This layer treats the sentence as a fully connected graph. It calculates relations between every pair of words, allowing information to flow directly across the entire sequence regardless of distance.
- MLP Layer: A multi-layer perceptron refines these relational features for the attention component.
By utilizing residual connections, the RNL produces a rich, "refined" representation of the source text that captures both local and global dependencies.
Experimental Setup
The model (RNMT) was evaluated on several benchmarks:
- NIST (Chinese-English): 1.25 million training pairs, tested on MT03-MT06.
- WMT17 (English-German): A large-scale dataset with 5.6 million pairs.
- WMT16 (English-German): Used for comparison against state-of-the-art syntactic models.
Key Results
The RNMT model demonstrated significant improvements in translation quality:
- BLEU Score Gains: On the NIST task, the model achieved an average BLEU score increase of 1.48 points. On WMT17, it saw a gain of 1.7 points.
- Robustness to Sentence Length: One of the most notable findings was the model's stability as sentence length increased. While standard RNN-based models often see a sharp decline in performance for long sentences, RNMT maintained high BLEU scores even for sequences exceeding 50 words.
- Superiority over Syntactic Models: RNMT outperformed models like BiRNN+GCN, which rely on supervised syntactic knowledge. This suggests that the relationships learned automatically by the RNL are highly effective for translation tasks.
- Improved Alignment: The Alignment Error Rate (AER) decreased by 1.1 points, indicating that the refined representations help the attention mechanism focus more accurately on relevant source words.
Conclusion and Future Work
The Relation Network Layer provides a powerful, data-driven way to model word relationships and mitigate the forgetting problem in NMT. While the $O(n^2)$ complexity of all-to-all relations is a consideration for very long sequences, the performance gains are substantial. Future research may explore integrating RNL with the Transformer architecture or developing more efficient graph pruning techniques to focus only on the most critical word relationships.
Summary updated