TL;DR
This paper provides a comprehensive empirical comparison of different text granularities—word, subword, character, and hybrid models—in Chinese-English Neural Machine Translation (NMT). The authors demonstrate that while subword models (like BPE) are generally superior for Chinese-to-English translation, hybrid word-character models take the lead for English-to-Chinese tasks.
What Problem Does This Paper Address?
Most NMT systems operate with a fixed vocabulary. When a system encounters a "rare word" not in its list, it typically replaces it with a generic <UNK> (unknown) symbol. This leads to a loss of information and poor translation quality.
This "open-vocabulary" problem is particularly acute in Chinese-English translation. Chinese lacks explicit word boundaries (no spaces), making word segmentation an error-prone preprocessing step. Conversely, English has a massive vocabulary due to morphology (prefixes, suffixes, etc.). The authors set out to find the "Goldilocks" level of granularity—not too big (words), not too small (characters), but just right—to maximize translation accuracy (BLEU scores) for this specific language pair.
Methods
The authors use a standard attention-based encoder-decoder NMT architecture (specifically, a stacked LSTM model) as their baseline. They then systematically test several ways to "tokenize" or break down the input and output text:
- Word Level: The traditional approach using standard segmentation tools.
- Character Level: Breaking Chinese sentences into individual characters.
- Hybrid Word-Character: Keeping frequent words intact but breaking rare words into characters. They use special prefixes (
<B>,<M>,<E>) to denote the position of a character within a word so the process can be reversed. - Subword Level: Using data-driven compression algorithms to find common character sequences. They compare two popular methods:
- Byte Pair Encoding (BPE): Iteratively merges the most frequent adjacent pairs of characters/sequences.
- Wordpiece Model (WPM): A similar approach used in systems like Google's NMT, which maximizes the likelihood of the training data.
Datasets
The study utilizes a substantial corpus for training and evaluation:
- Training Data: 2.09 million sentence pairs extracted from various LDC (Linguistic Data Consortium) corpora.
- Evaluation (Chinese-to-English): NIST 2003 (validation), and NIST 2004, 2005, and 2006 (test sets).
- Evaluation (English-to-Chinese): NIST 2003 (validation) and NIST 2008 (test set).
Key Insights and Ideas
- Subwords are Robust to Vocab Size: Unlike word-level models, which see a sharp drop in performance when the vocabulary is small (e.g., 15k), subword models like WPM and BPE maintain high performance. This suggests they are much more efficient at representing meaning with fewer symbols.
- Directionality Matters: The optimal strategy for translating into English isn't the same as translating into Chinese. Subwords excel for English targets, likely because they handle English morphology well. Hybrid models excel for Chinese targets, likely because they balance the semantic density of Chinese words with the flexibility of characters.
- The "Hybrid BPE" Sweet Spot: The authors found that using a Hybrid Word-Character approach on the source side and BPE on the target side yielded the absolute best results for Chinese-to-English translation.
- Character-Level Weakness: Pure character-level models performed poorly on very short sentences (under 20 tokens). The authors hypothesize that individual characters lack enough context to represent the meaning of short, pithy sentences effectively.
Key Results
- Chinese-to-English (30k Vocab): BPE achieved the highest average BLEU score of 43.75, compared to the Word-level baseline of 42.11.
- Chinese-to-English (15k Vocab): WPM reached 43.72 BLEU, significantly outperforming the Word-level model's 39.97.
- English-to-Chinese: The Hybrid Word-Character model performed best with a BLEU score of 22.36, followed closely by WPM at 22.14.
- Long Sentence Performance: Subword models (specifically WPM) showed a significant advantage in translating very long sentences (60-80+ words) compared to word-level or pure character-level models.
Limitations and Future Work
The authors noted a surprising "diminishing return" when increasing vocabulary size. For subword models, moving from a 30k vocabulary to a 60k vocabulary actually decreased performance. They infer that adding too many fine-grained subword units might introduce noise or make the model harder to train.
Future research directions suggested by the work include:
- Investigating these granularities in extremely low-resource settings.
- Exploring if these findings hold true for other "logographic" languages (like Japanese) paired with alphabetic ones.
- Refining the "Hybrid BPE" approach to see if it can be further optimized for bidirectional translation.
Summary updated