TL;DR
This paper investigates how different ways of breaking down Korean text (tokenization) affect the SacreBLEU metric's ability to predict human judgments of machine translation quality. They found that no single tokenization strategy works best for all metrics, and that some strategies, particularly character-level ones, can be surprisingly effective and efficient for Korean.
What Problem Does This Paper Address?
Machine translation (MT) evaluation is crucial for developing better MT systems. While BLEU has been the standard for years, its limitations have led to the adoption of more robust metrics like SacreBLEU. SacreBLEU aims to standardize MT evaluation by incorporating text normalization, including tokenization, to reduce variability. However, for morphologically rich, agglutinative languages like Korean, where words are formed by combining many morphemes, standard tokenization methods (like splitting by spaces) are insufficient. This can lead to inconsistent and unreliable evaluation scores, making it difficult to trust the metric's correlation with human judgment. The paper addresses this by systematically examining how various Korean tokenization strategies impact the human correlation of SacreBLEU and its constituent metrics (BLEU, TER, ChrF).
Methods
The researchers conducted a meta-evaluation study. They used SacreBLEU to evaluate machine translations from four commercial MT systems. The core of their methodology involved applying eleven different tokenization schemes to the Korean text:
- Word: Standard whitespace splitting.
- Morpheme: Various Korean morphological analyzers (Kkma, Hannanum, Komoran, MeCab, Okt, Kiwi, Khaiii).
- Subword: SentencePiece (SPM).
- Character: Treating each character as a token.
- CV (Consonant-Vowel): The smallest unit, Jamo, representing individual consonants and vowels.
They then computed the Pearson correlation coefficient between the automatic metric scores and human judgments (obtained through Direct Assessment) at both the segment and corpus levels. This allowed them to quantify how well each tokenization strategy aligned with human perception of translation quality. They also analyzed the computational cost of each tokenization method.
Datasets
The paper used the WMT 2020 English III-type test set for their source texts. For reference translations, they hired professional translators. The system translations were generated by four anonymized online MT models. Human evaluation was conducted using Direct Assessment (DA) on 2,048 sentences.
Key Insights and Ideas
- Tokenization is Not One-Size-Fits-All: The study strongly suggests that a "one-size-fits-all" tokenization approach is suboptimal for evaluating Korean MT. Different metrics (BLEU, TER, ChrF) respond differently to various tokenization granularities.
- Character-Level Can Be Surprisingly Good: For segment-level evaluation, character-level tokenization often yielded good human correlation for BLEU and TER, and was also computationally efficient. This challenges the intuition that more linguistic granularity is always better.
- Morpheme-Level Dominates Corpus-Level: While character-level was effective at the segment level, morpheme-level tokenization (especially with specific analyzers like Kiwi and Khaiii) showed higher correlation with human judgment at the corpus level.
- CV Level's Downside: The smallest unit, CV (consonant-vowel), while potentially useful for some metrics, often led to poor human correlation and, critically, had prohibitively high computational costs.
- Raw Scores are Not Comparable: The raw scores of metrics can be significantly inflated (up to double) depending on the tokenization used. This means directly comparing SacreBLEU scores across studies without considering their tokenization is misleading.
- System Rankings Can Flip: The choice of tokenization can even alter the ranking of MT systems, leading to erroneous conclusions about system performance.
Key Results
- Segment Level:
- BLEU and TER benefited from finer-grained tokenization, with character-level showing strong correlation (r=0.312 for BLEU).
- ChrF's correlation was more consistent across token types, but morpheme-level was generally best.
- CV tokenization resulted in poor correlation for TER and ChrF.
- Corpus Level:
- Morpheme-level tokenization generally achieved much higher correlations (ranging from 0.46 to 0.93).
- Kiwi and Khaiii were particularly strong morpheme tokenizers at the corpus level.
- Character tokens or SPM degraded TER and ChrF correlation at the corpus level, a contrast to segment-level findings.
- Computational Cost: TER could be up to seven times slower than other metrics, and when combined with CV, it took 360 times longer than BLEU for computation. Character-level tokenization was significantly faster than Kkma.
Limitations and Future Work
The authors acknowledge that the number of MT systems evaluated was small, which might limit the generalizability of corpus-level conclusions. They also note that their study was limited to Korean and future work should explore other agglutinative languages. They suggest scaling up the experiment with state-of-the-art NMT models and investigating the influence of token types on other comparable languages.
Summary updated