Abstract
Neural machine translation (NMT) has proven to outperform conventional statistical machine translation (SMT) across various language pairs. However, NMT systems typically operate with a fixed vocabulary, leading to an "open-vocabulary" problem where rare words are replaced by unknown symbols. This study conducts an extensive comparison of translation granularities—word, subword, character, and hybrid—specifically for Chinese-English NMT. Our experiments demonstrate that subword models perform best for Chinese-to-English translation with moderate vocabulary sizes, while hybrid word-character models are most suitable for English-to-Chinese translation. Notably, the Hybrid_BPE method achieves the best results on the Chinese-to-English task.
Introduction
Most NMT systems encounter a bottleneck when dealing with "rare words" not present in their fixed vocabulary. These words are often replaced with a generic <UNK> symbol, causing significant information loss and poor translation quality. This issue is particularly challenging in Chinese-English translation due to the lack of explicit word boundaries in Chinese and the complex morphology of English. This paper investigates the optimal granularity—balancing semantic density and vocabulary flexibility—to maximize translation accuracy (BLEU scores) for this specific language pair.
Methods
The study utilizes a standard attention-based encoder-decoder NMT architecture with stacked LSTMs. The core of the investigation lies in comparing different tokenization strategies to find the "Goldilocks" level of granularity:
- Word Level: The traditional approach using standard segmentation tools.
- Character Level: Breaking Chinese sentences into individual characters.
- Hybrid Word-Character: Keeping frequent words intact but breaking rare words into characters. Special prefixes (
<B>,<M>,<E>) denote the position of a character within a word to allow for reversible processing. - Subword Level: Using data-driven compression algorithms like Byte Pair Encoding (BPE), which merges frequent adjacent sequences, or the Wordpiece Model (WPM), which maximizes training data likelihood.
Key Results and Insights
The experiments, conducted on 2.09 million sentence pairs from LDC corpora, revealed several critical findings regarding the impact of granularity on translation performance:
- Subword Robustness: Subword models (BPE and WPM) are more robust to vocabulary size constraints than word-level models. At a 15k vocabulary size, WPM reached 43.72 BLEU, significantly outperforming the word-level model's 39.97.
- Directionality Matters: Subwords excel when translating into English, likely due to their ability to handle English morphology. Conversely, hybrid models are superior for Chinese targets, balancing word-level semantics with character-level flexibility.
- Performance on Long Sentences: Subword models showed a marked advantage in translating long sentences (60-80+ words) compared to pure word or character-level approaches.
As shown in the results table, the BPE method achieved the highest average BLEU score of 43.75 for Chinese-to-English translation at a 30k vocabulary size, while the Wordpiece method achieved the highest score on the MT03 development set.
Conclusion
This empirical study highlights that there is no "one-size-fits-all" granularity for NMT. While subword models like BPE are highly effective for English targets, hybrid models offer advantages for Chinese. Interestingly, increasing vocabulary size beyond a certain point (e.g., from 30k to 60k for subwords) can lead to diminishing returns, suggesting that excessive granularity may introduce noise. Future work will explore these strategies in low-resource settings and other logographic language pairs.
Summary updated