Abstract
Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant method to tackle this problem is Byte Pair Encoding (BPE) which splits words, including OOV words, into sub-word segments. BPE has achieved impressive results for a wide range of translation tasks in terms of automatic evaluation metrics. While it is often assumed that by using BPE, NMT systems are capable of handling OOV words, the effectiveness of BPE in translating OOV words has not been explicitly measured. In this paper, we study to what extent BPE is successful in translating OOV words at the word-level. We analyze the translation quality of OOV words based on word type, number of segments, cross-attention weights, and the frequency of segment n-grams in the training data. Our experiments show that while careful BPE settings seem to be fairly useful in translating OOV words across datasets, a considerable percentage of OOV words are translated incorrectly. Furthermore, we highlight the slightly higher effectiveness of BPE in translating OOV words for special cases, such as named-entities and when the languages involved are linguistically close to each other.
Introduction
In the field of Neural Machine Translation (NMT), "vocabulary sparsity" remains a persistent hurdle. Regardless of the training set size, models inevitably encounter Out-Of-Vocabulary (OOV) words at test time. The industry-standard solution is Byte Pair Encoding (BPE), a sub-word segmentation technique that breaks rare words into common sub-units (e.g., "unfriendly" becomes "un@@", "friend", "@@ly").
Because BPE ensures that the model almost never encounters an "unknown" token, the research community has largely assumed that the model is successfully translating these words. However, this paper from researchers at the University of Amsterdam provides a necessary "reality check," investigating whether the ability to process sub-units actually translates to the ability to convey meaning at the word level.
Methodology
To evaluate BPE's effectiveness, the researchers moved beyond aggregate automatic metrics and conducted a granular, manual evaluation. Their approach involved several key steps:
- Manual Annotation: The team randomly sampled 400 unique OOV words from three language pairs: German-English (De-En), Russian-English (Ru-En), and Romanian-English (Ro-En).
- Categorization: Native speakers classified these OOV words into specific types: Named Entities (NE), Compounds (C), Morphological Variants (MV), Spelling Errors (SE), Technical words (T), and Foreign words (F).
- Quality Labeling: Translations were manually assessed as Correct, Partly Correct (e.g., minor morphological or spelling errors), or Wrong.
- Alignment via Attention: Using cross-attention weights from the Transformer model, the researchers identified word-level correspondences to determine exactly how an OOV word was represented in the output.
- Hyperparameter Analysis: They compared different BPE "merge operations" (10k vs. 37k) to determine if the granularity of sub-words influenced OOV accuracy.
Key Insights and Analysis
The "Unknown" Illusion
While BPE technically eliminates the [UNK] token, it does not eliminate translation errors. The study found that as the density of OOV words in a sentence increases, the overall translation quality—measured by human Direct Assessment scores—drops significantly. BPE provides a way for the model to produce something, but that output is frequently incorrect.
Granularity and Linguistic Proximity
One of the more counter-intuitive findings is that using a smaller number of BPE merge operations (resulting in more, smaller sub-word segments) actually improves OOV translation. Smaller segments appear to give the model a better chance to reconstruct the meaning of a rare word from its constituent parts.
Furthermore, the effectiveness of BPE is heavily dependent on the linguistic distance between languages. BPE performs significantly better when the source and target languages are linguistically similar (such as German and English) compared to more distant pairs like Russian and English.
Performance by Word Type
Translation success varies drastically depending on the nature of the OOV word:
- Named Entities: These are the most successfully translated (~85% in De-En), likely because they often involve simple character copying or transliteration.
- Compounds and Morphological Variants: These categories are much more challenging. In German-English, correct translation for compounds dropped to approximately 56%. Morphological variants—different forms of a known root word—showed the lowest rates of correct translation, suggesting BPE struggles to capture the nuances of complex word structures.
The Role of Attention and N-grams
The researchers found a strong correlation between cross-attention weights and translation quality; correct translations consistently received higher attention weights than incorrect ones. Interestingly, the frequency of a sub-word's n-grams in the training data did not strongly correlate with success, suggesting that the model's ability to generalize is not merely a matter of simple pattern matching.
Conclusion and Future Directions
The study concludes that across datasets, BPE-based models only manage to translate roughly 60% of OOV words correctly or partly correctly. This highlights a "blind spot" in automatic metrics like BLEU, which often neglect these local, word-level failures that human readers find jarring.
Future work in NMT should move toward morphology-aware segmentation. Rather than relying on purely statistical approaches like BPE, models may need to understand the linguistic structure of words, particularly for morphologically rich languages. Additionally, there is a clear need for new automatic metrics that can specifically target and evaluate OOV accuracy without the need for intensive manual annotation.
Summary updated