Abstract
Unseen words, also called out-of-vocabulary words (OOVs), are difficult for machine translation. In neural machine translation, byte-pair encoding can be used to represent OOVs, but they are still often incorrectly translated. We improve the translation of OOVs in NMT using easy-to-obtain monolingual data. We look for OOVs in the text to be translated and translate them using simple-to-construct bilingual word embeddings (BWEs). In our MT experiments we take the 5-best candidates, which is motivated by intrinsic mining experiments. Using all five of the proposed target language words as queries we mine target-language sentences. We then back-translate, forcing the back-translation of each of the five proposed target-language OOV-translation-candidates to be the original source-language OOV. We show that by using this synthetic data to fine-tune our system the translation of OOVs can be dramatically improved. In our experiments we use a system trained on Europarl and mine sentences containing medical terms from monolingual data.
Introduction
Neural Machine Translation (NMT) has significantly advanced the state of the art, yet it continues to struggle with "rare words" or Out-of-Vocabulary (OOV) terms. While techniques like Byte-Pair Encoding (BPE) prevent the system from encountering unknown strings by decomposing words into sub-units, they do not guarantee semantic accuracy. Specialized terminology, such as medical jargon, is frequently mistranslated because it is underrepresented in standard parallel training corpora like Europarl.
Traditional approaches to this problem include "pointing" (copying the source word) or post-processing with external dictionaries. However, copying is ineffective for terms that differ across languages, and dictionary-based post-processing often fails to account for sentence context. This paper proposes a method to dynamically teach NMT models new terminology using bilingual word embeddings and synthetic data generation.
Methodology
The proposed pipeline identifies OOVs in the source text and integrates them into the NMT model through the following steps:
- OOV Identification: Identifying terms in the source text that are absent from the initial parallel training data.
- Bilingual Word Embeddings (BWEs): Utilizing BWEs to map source and target words into a shared vector space. The retrieval of the top $n$ translation candidates is enhanced by incorporating orthographic similarity, which leverages the shared Greek or Latin roots common in technical domains.
- Terminology Mining: A large target-language monolingual corpus is searched for sentences containing the top 5 translation candidates.
- Forced Back-Translation: Mined target sentences are back-translated into the source language. Crucially, the translation of the candidate word is forced to match the original source OOV, creating a synthetic parallel corpus where the OOV appears in a natural linguistic context.
- Fine-Tuning: The NMT model undergoes a single fine-tuning step on this synthetic data, allowing it to learn the new term's usage without catastrophic forgetting of its prior training.
Experimental Setup
The baseline NMT system was trained on Europarl v7 (approx. 1.9M English-German sentence pairs). Medical terminology was mined from the UFAL Medical Corpus (3.1M sentences). Evaluation was conducted on the HimL (Health In My Language) test sets, specifically the Cochrane and NHS24 portions, which represent real-world medical translation challenges.
Results
The integration of mined terminology led to substantial improvements in translation quality.
Quantitative Analysis
As shown in Figure 1, the fine-tuning approach significantly outperformed the baseline and simple copying methods. On the Cochrane test set, the BLEU score increased from 22.4 to 27.2, a gain of 4.8 points. The NHS24 test set saw a 2.3-point increase, from 20.2 to 22.5.
Qualitative Analysis
The effectiveness of the method is further evidenced by qualitative improvements. The baseline model often produced unrelated words or nonsense when encountering medical OOVs. In contrast, the fine-tuned model correctly translated terms like "nosebleeds" to Nasenbluten and "lightheadedness" to Schwächegefühl.
Conclusion
A key finding of this research is the effectiveness of using the top 5 translation candidates rather than just the single best match. By providing multiple options within synthetic sentences, the fine-tuning process allows the NMT system to leverage its contextual understanding to select the most appropriate translation. Furthermore, the inclusion of orthographic similarity in BWE retrieval proved vital for the medical domain. While the method depends on the quality of BWEs and the availability of relevant target-language monolingual data, it offers a robust framework for domain adaptation.
Summary updated