TL;DR
This paper introduces a novel, efficient, and robust neural machine translation (NMT) model that leverages translation memory (TM). By using a single bilingual sentence as TM and a new joint training criterion, the model achieves state-of-the-art performance on TM-specialized tasks and significantly improves upon standard NMT on general tasks, while also being faster.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) has achieved remarkable success, but integrating Translation Memory (TM) – a database of previously translated sentence pairs – has been challenging. While TMs are known to be beneficial, existing TM-based NMT approaches often suffer from high computational overhead during training or inference, and their effectiveness on general translation tasks (where the TM might not be a perfect match for the input) is questionable. Some prior work has even failed to show improvements on standard benchmarks like WMT news translation. This paper aims to bridge this gap by proposing a TM-based NMT system that is both fast and accurate, works well on general tasks, and addresses a critical robustness issue.
Methods
The core of the proposed approach lies in a simplified TM-based NMT architecture within the Transformer framework.
- Light-weight TM Configuration: Instead of using multiple TM sentences, the model uses a single, most similar bilingual sentence pair as the TM for each source sentence. This significantly reduces computational complexity.
- TM Encoding: Three methods are proposed to encode the TM into a vector representation:
- TF-S (Sentence): Uses word and position embeddings of the TM's target sentence.
- TF-SS (Sentence with Score): Incorporates a similarity score between the source sentence and the TM's source sentence, scaling the TM's target sentence embedding.
- TF-SA (Sentence with Alignment): Leverages word alignment between the TM's source and target sentences to give more attention to relevant words. This method proved to be the most effective.
- Example Layer: A novel "Example Layer" is introduced in the decoder. This layer incorporates multi-head attention mechanisms that attend to both the previously generated target sequence and the encoded TM, allowing the model to effectively utilize TM information.
- Robustness-focused Training Criterion: A key contribution is a novel joint training criterion. This criterion trains the model on two datasets simultaneously: one with TM information and another where the TM is effectively nullified (using empty sentences). This "joint training" forces the model to learn to translate effectively both with and without TM guidance, addressing the robustness issue where performance degrades when the TM is not a good match.
Datasets
The paper evaluates its models on a variety of datasets:
- TM-specialized Tasks: JRC-Acquis corpora, covering six language pairs (English↔German, English↔Spanish, English↔French).
- General Tasks: WMT14 English-to-German (En→De) and WMT17 Chinese-to-English (Zh→En) tasks.
- The paper also mentions using a subset of the Es→En task for analyzing robustness based on TM similarity.
Key Insights and Ideas
- Single TM Sentence Sufficiency: The research demonstrates that using just the single most similar bilingual sentence as TM can be highly effective, challenging the assumption that more TM data is always better for performance.
- Robustness is Crucial: A significant finding is that standard TM-based NMT models can be brittle, performing poorly on inputs whose TMs are not highly similar. This highlights the importance of robustness for practical deployment.
- Joint Training as a Solution: The proposed joint training approach effectively addresses the robustness issue, enabling a single model to handle both TM-rich and TM-poor translation scenarios.
- Alignment-based TM Encoding is Powerful: The TF-SA method, which uses word alignment, yields the best results, suggesting that understanding which parts of the TM are most relevant is key.
- TM Benefits General Tasks: Contrary to previous findings, this work shows that TM-based NMT can indeed improve performance on general translation tasks, not just specialized ones.
Key Results
- TM-Specialized Tasks: The proposed TF-SA model, trained with the joint criterion, significantly outperforms strong baselines (like TF-P, TF-SEQ, FM+, TF-G) by up to 4.9 BLEU points. It also achieves substantial gains over standard Transformer (TF) by at least 1.2 BLEU points.
- General Tasks: The model achieves notable improvements on WMT En→De (0.7 BLEU points) and WMT Zh→En (1.0 BLEU point) tasks, outperforming baselines on average.
- Speed: The proposed model is significantly faster in both training and testing compared to many TM-based baselines, making it more practical for real-world applications. For instance, testing is more efficient than TF-P.
- Robustness: After joint training, the model shows substantial gains (1.2 BLEU points) over standard training on the entire test set and significantly outperforms the TF baseline on subsets with low TM similarity.
- Noisy TMs: The TF-SA model demonstrates robustness even to noisy TMs, performing only slightly worse than the noise-free version and still outperforming the best TM baseline.
Limitations and Future Work
- Robustness Issue with Standard Training: The paper clearly identifies that standard training of TM-based NMT models suffers from a robustness issue, performing worse than standard Transformer when TM similarity is low. The joint training criterion is presented as a solution.
- Future Work: The paper implicitly suggests that further exploration of TM encoding methods and potentially more sophisticated joint training strategies could lead to even greater improvements. The practical utility of the model for online services is highlighted, suggesting future work could focus on deployment and real-time performance optimization.
Summary updated