How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

Ali Araabi, Christof Monz, Vlad Niculae

Introduction

Neural Machine Translation (NMT) models face a significant challenge known as "vocabulary sparsity." These models typically only learn to translate words encountered during training. Words absent from the training set, termed Out-Of-Vocabulary (OOV) words, are prevalent in real-world scenarios, appearing in 15% to 60% of test sentences.

Byte Pair Encoding (BPE) has become the industry standard for addressing this. By decomposing rare or unseen words into subword units (e.g., "laryngitis" into "laryn" + "gitis"), BPE ensures the model can always generate an output. However, while BPE prevents "unknown token" errors, its actual effectiveness in producing correct translations at the word level remains under-scrutinized. This paper investigates whether BPE facilitates genuine translation or merely enables the model to "guess" based on subword patterns.

Methodology

To move beyond aggregate metrics like BLEU, which can obscure specific failures, the authors conducted a granular manual analysis:

  • Manual Annotation: Native speakers evaluated 400 unique OOV words across three language pairs, categorizing translations as Correct, Partly Correct, or Wrong.
  • OOV Categorization: Words were classified into types including Named Entities (NE), Compounds (C), Morphological Variants (MV), Spelling Errors (SE), Technical terms (T), and Foreign words (F).
  • Attention Analysis: Cross-attention weights from the Transformer's penultimate layer were used to map source OOV words to their corresponding target outputs.
  • Granularity Testing: The study compared different BPE merge settings (e.g., 10k vs. 37k operations) to assess the impact of subword size on translation accuracy.

Linguistic Context and Datasets

The research utilized standard WMT datasets (Europarl, Common Crawl, News Commentary) for three linguistically diverse pairs:

  • German-English (De-En): Characterized by complex compound words.
  • Russian-English (Ru-En): Highly morphologically rich Slavic language.
  • Romanian-English (Ro-En): Morphologically rich Romance language.

Key Results

The study reveals that while BPE is a useful tool, it is not a complete solution for the OOV problem.

Performance Degradation

As the OOV rate in a sentence increases, translation quality drops sharply. Automated metrics like BLEU reflect this trend, showing that models struggle as they encounter more unseen vocabulary.

BLEU score performance as OOV rate increases across different BPE vocabulary sizes.

Manual Assessment Findings

Manual annotation provides a sobering look at OOV translation success. On average, BPE-based models correctly translate only about 60% of OOV words.

Manual annotation of OOV translation quality across three language pairs.

Specific findings include:

  • The Morphology Trap: Morphological variants are particularly challenging. Even when a model "knows" a root word, it often fails to produce the correct inflected form in the target language.
  • Compound Failures: In German, only 56% of OOV compounds were translated accurately.
  • Named Entity Success: Conversely, BPE excels at Named Entities, likely due to high lexical similarity or transliteration patterns.
  • Vocabulary Size: Smaller BPE vocabularies (10k merges), which split words more aggressively, generally outperformed larger ones (37k) for OOV translation.

Discussion and Insights

The research highlights that linguistic distance significantly impacts BPE's effectiveness. It performs better between similar languages (German-English) than between distant ones (Russian-English). Furthermore, a strong correlation was found between translation quality and cross-attention weights; correctly translated OOVs typically receive higher attention, suggesting the model's internal confidence aligns with accuracy.

Conclusion

BPE effectively hides the OOV problem by ensuring every word can be processed, but it does not guarantee translation accuracy. The high failure rate for morphological variants and compounds suggests that statistical subword splitting is insufficient for linguistically complex languages. Future research should explore morphologically-aware segmentation techniques to bridge the gap between statistical frequency and linguistic structure.

Summary updated

Abstract

Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant method to tackle this problem is Byte Pair Encoding (BPE) which splits words, including OOV words, into sub-word segments. BPE has achieved impressive results for a wide range of translation tasks in terms of automatic evaluation metrics. While it is often assumed that by using BPE, NMT systems are capable of handling OOV words, the effectiveness of BPE in translating OOV words has not been explicitly measured. In this paper, we study to what extent BPE is successful in translating OOV words at the word-level. We analyze the translation quality of OOV words based on word type, number of segments, cross-attention weights, and the frequency of segment n-grams in the training data. Our experiments show that while careful BPE settings seem to be fairly useful in translating OOV words across datasets, a considerable percentage of OOV words are translated incorrectly. Furthermore, we highlight the slightly higher effectiveness of BPE in translating OOV words for special cases, such as named-entities and when the languages involved are linguistically close to each other.
BibTeX
@inproceedings{araabi2022how,
  author = {Ali Araabi and Christof Monz and Vlad Niculae},
  title = {{How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?}},
  booktitle = {AMTA},
  year = {2022},
  url = {https://openreview.net/forum?id=vqYS0hWOvZ9},
}
Figure