TL;DR
This paper introduces "jamo-level" subword tokenization for Korean machine translation, breaking down syllables into their constituent "jamo" (consonants and vowels). This approach leads to shorter sequences and more efficient models, particularly in low-resource settings, outperforming traditional syllable-level and byte-level tokenization.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) models rely heavily on subword tokenization to handle large vocabularies and unseen words. However, languages like Korean, with their vast number of unique syllables (over 11,000), pose a challenge. Representing each syllable as a distinct token requires a massive vocabulary, leading to increased model size and computational cost, especially in low-resource scenarios where data is scarce. Existing methods like syllable-level tokenization are limited by the syllable structure, while byte-level tokenization lacks linguistic grounding and can struggle with Korean's morphology. This paper aims to find a more efficient and linguistically informed tokenization strategy for Korean.
Methods
The core innovation is jamo-level subword tokenization. Korean syllables are inherently compositional, breaking down into 2-3 "jamo" (consonants and vowels), of which there are only 68. The authors propose using these jamo as the atomic units for Byte-Pair Encoding (BPE), a standard subword tokenization algorithm. This means instead of learning merges between syllables, BPE learns merges between jamo.
They explore two types of jamo representations:
- Positional Jamo: These maintain the distinct roles of initial consonants, vowels, and final consonants.
- Compatibility Jamo: These merge visually similar jamo, resulting in a smaller set of visually distinct units.
The paper also investigates the impact of subword regularization (specifically BPE-Dropout) in conjunction with jamo-level tokenization, hypothesizing that the increased tokenization flexibility offered by jamo would amplify its benefits.
Datasets
The experiments were conducted on two main translation tasks:
- English↔Korean: A subset of the AI-Hub News Translation corpus (200k sentence pairs) was used, with a focus on low-resource settings. A "restricted vocabulary" setting was also created, where the vocabulary size was limited to the number of unique characters in the syllable-level corpus.
- Korean↔Jeju-eo: A low-resource language pair, using a dataset of 180k sentence pairs.
Key Insights and Ideas
- Linguistic Grounding Pays Off: Decomposing Korean syllables into jamo provides a linguistically motivated representation that byte-level tokenization misses. This allows models to better capture sub-syllable morphology.
- Efficiency Gains are Significant, Especially When Small: At smaller vocabulary sizes, jamo-level BPE drastically reduces the number of tokens and vocabulary parameters compared to syllable-level BPE. This is crucial for fitting models into tight constraints.
- Subword Regularization Amplified: The increased flexibility in tokenizing sequences at the jamo level leads to a much larger space of possible tokenizations for subword regularization, enhancing model robustness.
- Positional Jamo Outperforms: Positional jamo generally yielded better results than compatibility jamo, suggesting that preserving the distinct roles of consonants and vowels is beneficial.
- Byte-Level BPE Struggles with Korean: Byte-level BPE performed poorly, especially in generating Korean, highlighting its inability to capture Korean's morphological nuances.
Key Results
- Improved Compression: Jamo-level BPE consistently produced shorter tokenized sequences than syllable-level BPE across various vocabulary sizes, especially at smaller sizes. For instance, at a vocabulary size of 1000, jamo-level models produced nearly 25-40% shorter sequences than syllable-level models.
- Translation Performance: In low-resource settings (Korean↔Jeju-eo and restricted English↔Korean), positional jamo-level models significantly outperformed syllable-level and byte-level models. For example, in Korean↔Jeju-eo, positional jamo-level models with dropout improved performance by roughly 0.6-1.0 BLEU over syllable and byte-level models.
- Dropout Benefits: Subword regularization (dropout) was more effective with jamo-level tokenization, particularly in low-resource scenarios, leading to larger performance gains than with syllable-level models.
- No Invalid Sequences: Despite concerns about generating invalid sequences (e.g., incorrect jamo order), the experiments showed no instances of invalid sequence generation after a short training period.
Limitations and Future Work
- Dataset Limitations: The study used only two datasets, with Korean↔Jeju-eo being a very closely related language pair. Comparing with other language families would be beneficial.
- Vocabulary Size Sweep: A large hyperparameter sweep for vocabulary size was not performed, and the chosen sizes were somewhat arbitrary. It's possible that different vocabulary sizes could yield different results.
- Pre-trained Models: The experiments did not use large, pre-trained models, which are common in modern NLP. Adapting jamo-level tokenization to these models would be a valuable next step.
- Scaling with Dataset Size: The benefits of jamo-level tokenization diminished as dataset size increased, suggesting that its primary advantage lies in low-resource and restricted-vocabulary settings. Future work could explore how jamo-level tokenization interacts with larger models and datasets.
- Alternative Tokenizers: The paper suggests that other tokenization algorithms like UnigramLM might better leverage the sub-syllable information captured by jamo.
Summary updated