Abstract
For different language pairs, word-level neural machine translation (NMT) models with a fixed-size vocabulary suffer from the same problem of representing out-of-vocabulary (OOV) words. The common practice usually replaces all these rare or unknown words with a token, which limits the translation performance to some extent. Most of recent work handled such a problem by splitting words into characters or other specially extracted subword units to enable open-vocabulary translation. Byte pair encoding (BPE) is one of the successful attempts that has been shown extremely competitive by providing effective subword segmentation for NMT systems. In this paper, we extend the BPE style segmentation to a general unsupervised framework with three statistical measures: frequency (FRQ), accessor variety (AV) and description length gain (DLG). We test our approach on two translation tasks: German to English and Chinese to English. The experimental results show that AV and DLG enhanced systems outperform the FRQ baseline in the frequency weighted schemes at different significant levels.
Introduction
Neural Machine Translation (NMT) models typically operate with a fixed-size vocabulary, which introduces the significant hurdle of Out-of-Vocabulary (OOV) words. When a model encounters a rare word not present in its dictionary, it typically defaults to a generic <UNK> token, resulting in a loss of semantic information. Subword segmentation—breaking words into smaller, more manageable units—has emerged as a standard solution. While Byte Pair Encoding (BPE) is the industry standard, its reliance solely on frequency (FRQ) to determine merge operations may not always capture the most linguistically meaningful units. This research proposes a generalized framework that incorporates advanced statistical measures to optimize subword boundaries.
Methodology
The proposed generalized BPE algorithm maintains the iterative bottom-up merging logic of standard BPE but introduces alternative "goodness measures" for selection:
- Frequency (FRQ): The standard BPE baseline.
- Accessor Variety (AV): Adapted from Chinese word segmentation, this measure evaluates the variety of characters appearing to the left and right of a substring. High variety indicates a stable unit likely to exist at a linguistic boundary.
- Description Length Gain (DLG): An information-theoretic approach that selects merges based on their ability to reduce the overall description length (bits) of the corpus, prioritizing compression efficiency.
To mitigate noise in large corpora, the authors implemented Frequency Weighted Schemes (denoted with a prime, e.g., AV'-BPE), ensuring that subwords derived from high-frequency, reliable words are prioritized during the segmentation process.
Experimental Results
The framework was evaluated on German-to-English (IWSLT 2014) and Chinese-to-English (WMT 2017) tasks. The results demonstrate that the proposed measures consistently outperform the standard frequency-based baseline.
As shown in Figure 1, the performance of all methods generally peaks around 20K merge operations. Notably, the frequency-weighted AV'-BPE and DLG'-BPE consistently maintain higher BLEU scores than the FRQ'-BPE baseline across most merge counts. In the German-English task, AV'-BPE achieved a peak BLEU score of 29.46, representing a significant improvement over the baseline.
Table 1 provides a granular look at these improvements. For the German-English task, the AV'-BPE method with beam search reached 29.46 at 20K merges, marked with high statistical significance (++) over the baseline. Similar trends were observed in Chinese-to-English tasks, where DLG-BPE outperformed the baseline by 0.65 BLEU points.
Key Insights and Discussion
- Compactness and Efficiency: Analysis revealed that AV and DLG measures tend to produce shorter sequences and preserve original word forms more effectively than frequency-based BPE at the same number of merge operations. This "compactness" is beneficial for NMT models, which often struggle with excessively long sequences.
- Language Sensitivity: The optimal segmentation measure appears to be language-dependent. AV performed exceptionally well for the morphologically rich German language, while DLG proved superior for the ideographic structure of Chinese. This suggests that subword segmentation should be tailored to the specific linguistic properties of the source and target languages.
Conclusion
By moving beyond simple frequency counts to include measures like Accessor Variety and Description Length Gain, this generalized framework offers a more robust approach to subword segmentation. The resulting improvements in BLEU scores across different language pairs highlight the potential for statistically-driven segmentation to enhance the performance of open-vocabulary NMT systems.
Summary updated