TL;DR
This paper proposes a novel, simple method to integrate translation memories (TMs) into Neural Machine Translation (NMT) systems without altering the NMT model's architecture or retraining it. By treating TMs as "prompts" during decoding, the system achieves significant improvements in translation quality, especially for repetitive content, by leveraging existing, high-quality bilingual data.
What Problem Does This Paper Address?
Machine translation (MT) systems, particularly Neural Machine Translation (NMT), struggle with highly repetitive content, such as technical manuals or legal documents, where consistent translation of specific phrases is crucial. Translation memories (TMs), which store previously translated sentence pairs, offer a valuable resource for improving translation quality in such scenarios. However, existing methods for integrating TMs into NMT often require substantial model modifications, retraining, or complex architectures, making them difficult to implement in real-world, production-ready systems. This paper addresses the challenge of efficiently and effectively incorporating TMs into NMT systems without these significant overheads, aiming to make TM-assisted NMT more accessible.
Methods
The core of this paper's contribution lies in its innovative approach to TM integration during the NMT decoding phase. Instead of modifying the NMT model itself, the authors treat retrieved TM sentence pairs as "prompts."
Here's a breakdown of their method:
- Prompting with TMs: For a given input sentence and a retrieved TM pair (source TM, target TM), the source TM is concatenated with the input sentence. This combined sequence is fed into the NMT encoder.
- Forced Decoding: On the decoder side, the target TM is generated first in a "forced" manner, meaning the model is compelled to output these tokens exactly. This is achieved by setting the probability of these tokens to 1.
- Automatic Generation: After the target TM is generated, the NMT model continues to generate the rest of the translation automatically, producing the final hypothesis.
- Concatenation Templates: To effectively combine the input sentence and TM, the authors design various "concatenation templates." These templates ensure that the NMT model can process the combined input and that the semantic integrity of each part is maintained. Templates include direct concatenation, using commas, semicolons, conjunctions, or enclosing TMs in parentheses.
- Fragment-Level TMs: The method also extends to fragment-level TMs. This involves identifying common fragments between the input sentence and the source TM, aligning them with corresponding tokens in the target TM, and then constructing fragment-level prompts.
This approach is novel because it leverages the power of pre-trained NMT models without requiring any changes to their architecture or retraining, making it highly practical.
Datasets
The authors evaluated their method on several datasets:
- TM Specialized Translation Tasks:
- DGT-TM: The entire body of European legislation, used for German-English (En-De, De-En).
- United Nations Parallel Corpus (UNPC): Resolutions from the UN General Assembly, used for English-Chinese (En-Zh), Russian-Chinese (Ru-Zh), and French-Chinese (Fr-Zh).
- Homemade English-Chinese dataset (H-m): A custom dataset of 3401 sentences with similar TM pairs.
- Domain Adaptation Task:
- Multi-domain datasets from Aharoni and Goldberg (2020): German-English bilingual datasets in five domains: Medical, Law, IT, Koran, and Subtitles.
For these datasets, they randomly selected 3,000 sentence pairs for testing, using the rest as TM databases. They also used publicly available pre-trained NMT models from Facebook (WMT19 De-En, En-De) and NiuTrans (WMT20 En-Zh).
Key Insights and Ideas
- TMs as Prompts: The central idea of treating TMs as prompts for NMT decoding is a significant departure from traditional TM integration methods. It allows for leveraging existing NMT models without modification, making TM integration much more accessible.
- Few-Shot Learning Analogy: The method aligns with the principles of few-shot learning, where a model can adapt to new tasks or data with minimal examples. Here, the TM acts as a few examples to guide the NMT.
- Simplicity and Efficiency: The method's strength lies in its simplicity. By only modifying the decoding process and using simple concatenation, it avoids the computational and engineering costs of retraining or architectural changes.
- Impact of TM Similarity: The effectiveness of the method is highly dependent on the similarity between the retrieved TM and the input sentence. High similarity yields substantial improvements, while low similarity can even degrade performance. This suggests a need for a similarity threshold.
- NMT Model "Strength" Matters: Stronger NMT models (trained on more data or with more complex architectures) benefit more from TM prompting, showing greater BLEU score improvements. Weaker models show less improvement or can even be negatively impacted by noisy TMs.
- Sentence Length Limitation: The method struggles with very long input sentences. Concatenating a long sentence with its TM can push the combined input beyond the typical sentence length distribution the NMT model was trained on, leading to performance degradation.
Key Results
- Significant BLEU Score Improvements: The proposed method achieved notable improvements in BLEU scores across various datasets. For instance, on the DGT-TM De-En test set, BLEU scores increased by up to 8.63 points.
- Comparable to State-of-the-Art: The method demonstrated comparable performance to existing state-of-the-art TM integration techniques, but with significantly less complexity.
- Effectiveness on Repetitive Content: The improvements were particularly pronounced on datasets with repetitive content, highlighting the practical utility of the approach.
- Domain Adaptation Success: In domain adaptation tasks, the method showed improvements in most domains, though slightly less than dedicated methods like kNN-MT, but with a significant advantage in speed and storage.
- Commercial System Applicability: The method was successfully implemented on a commercial NMT system (NiuTrans Enterprise), demonstrating its real-world applicability and yielding improvements of up to 3.55 BLEU points.
Limitations and Future Work
The authors acknowledge several limitations:
- Low Similarity TMs: The method's performance degrades when the retrieved TM has low similarity to the input sentence, potentially introducing noise.
- Long Sentences: The method struggles with very long input sentences due to deviations from the model's training distribution.
- Word Aligner Performance: For fragment-level TMs, the performance is dependent on the accuracy of the word alignment tools used.
Future work directions suggested by the authors include:
- Developing more effective methods to handle TMs with low similarity.
- Investigating techniques to alleviate the performance drop for long sentences.
- Exploring ways to improve the robustness of the method to noisy TM data.
Summary updated