TL;DR
This paper introduces a novel approach to Neural Machine Translation (NMT) for logographic languages like Chinese and Japanese by leveraging sub-character level information (ideographs and strokes). This method significantly improves translation quality by enabling more efficient representation of characters and better handling of rare words, outperforming traditional character-based NMT systems.
What Problem Does This Paper Address?
Modern NMT systems, particularly those using sequence-to-sequence models with attention mechanisms and sub-word units, have achieved remarkable success. However, these advancements have largely focused on alphabetic languages. Logographic languages, which use characters composed of ideographs and strokes rather than letters, present unique challenges. Traditional sub-word units in these languages often reduce to individual characters, failing to capture the rich structural and semantic information within them. This oversight leads to issues like the out-of-vocabulary (OOV) problem and less efficient translation, especially when dealing with languages that share similarities but have distinct writing systems. This paper aims to bridge this gap by exploring how sub-character level information can enhance NMT for logographic languages.
Methods
The core of this research lies in decomposing logographic characters into smaller, meaningful units: ideographs and strokes.
- Sub-character Unit Creation: The authors created a sub-character database by decomposing Chinese characters into ideographs (semantic and phonetic components) and further breaking down ideographs into strokes. They utilized existing resources like the CNS11643 charset for character decomposition and manually transcribed stroke data.
- NMT Model Architecture: They employed a standard sequence-to-sequence model with an attention mechanism, using Long Short-Term Memory (LSTM) units.
- Data Granularity: Experiments were conducted using various levels of data granularity: word-level, character-level, ideograph-level, and stroke-level.
- Shared Vocabularies: The study investigated the impact of shared vocabularies between source and target languages when they used the same sub-character level data.
- Post-tagging: To distinguish between identical ideograph or stroke sequences that might arise from different characters or words, a post-tagging mechanism was used.
Datasets
The experiments utilized the following datasets:
- ASPEC Corpus: This corpus includes a Japanese-English paper abstract corpus (ASPEC-JE) and a Japanese-Chinese paper excerpt corpus (ASPEC-JC).
- Casia2015 Corpus: This corpus contains approximately 1 million parallel Chinese-English sentences.
For training, the first million parallel sentences from ASPEC-JE and all of ASPEC-JC were used. For Chinese-English pairs, the entire Casia2015 corpus was utilized. For development and testing, random selections of 1,000 sentences were made from the development and test sets of Casia2015, as it lacked an official split.
Key Insights and Ideas
- Ideographs and Strokes as Building Blocks: Decomposing characters into ideographs and strokes creates smaller, more concentrated vocabularies, which is crucial for efficient NMT, especially for handling rare words.
- Language-Specific Preferences: Different logographic languages show distinct preferences for sub-character granularity. Chinese benefited more from ideograph-level information, while Japanese saw greater improvements with stroke-level data. This highlights the importance of tailoring sub-character units to the specific linguistic characteristics.
- Shared Information Amplifies Benefits: When source and target languages share similar writing systems and utilize the same sub-character level data, shared vocabularies further enhance translation performance.
- Addressing OOV and Grammaticality: Sub-character based NMT models effectively handle out-of-vocabulary (OOV) proper nouns and can produce more grammatically correct translations by better encoding source sentence nuances.
Key Results
- Logographic-Alphabetic Translation: Sub-character level models (ideograph and stroke) consistently outperformed baseline word and character-level models in Japanese/English and Chinese/English translation. For Japanese-English, stroke models were superior, while for Chinese-English, ideograph models performed better.
- Logographic-Logographic Translation (Chinese-Japanese):
- Character-level baselines ("char" and "bpe") outperformed "word" models.
- Sub-character level models showed significant improvements.
- When both source and target used sub-character data, ideograph models generally outperformed stroke models.
- The best performance was achieved by models using Chinese ideograph data and Japanese stroke data, yielding BLEU scores of 33.8 for Japanese-Chinese and 43.9 for Chinese-Japanese translation.
- Translation Example: The paper demonstrates a case where a proper noun ("Matsushita Electric") was handled correctly by the sub-character based model, avoiding the
<unk>symbol seen in word-based baselines, and producing a more natural translation.
Limitations and Future Work
- Stroke Model Vocabulary Size: The authors note that stroke models have a significantly smaller vocabulary (around 30 units) compared to ideograph models (over 500 units). This disproportionate size might explain why stroke models performed less well, especially when shared vocabularies were applied, and suggests that hyperparameter tuning might need to be adjusted for different granularities.
- Structural Information: Current methods of transforming characters into ideographs and strokes do not explicitly consider the structural information of how these components are combined. This can lead to data repetition, necessitating post-tagging. Future work could explore encoding this structural information directly into the data.
- Further Investigation: The paper suggests further investigation into why performance drops when shared vocabularies are applied to stroke models and why there are performance differences between ideograph and stroke data.
Summary updated