DBMS-KU Interpolation for WMT19 News Translation Task

Sari Dewi Budiwati, Al Hafiz Akbar Maulana Siagian, Tirana Noor Fatyanosa, Masayoshi AritsugiACL 2019

TL;DR

This paper tackles the challenging Kazakh-English translation task, a low-resource language pair with differing writing systems. The authors propose an interpolation system that combines direct translation with a pivot language (Russian) and explores the impact of different language model orders (3-gram vs. 5-gram) on translation quality, achieving improved BLEU scores over a baseline.

What Problem Does This Paper Address?

Machine translation (MT) systems often struggle with low-resource language pairs, where limited parallel data and morphological complexity hinder performance. Kazakh-English is a prime example, further complicated by its Cyrillic script versus English's Latin alphabet, which poses challenges for tokenization and segmentation. This paper aims to improve Kazakh-English translation by leveraging a pivot language strategy and investigating the influence of language model (LM) order on translation quality.

Methods

The core of the proposed system is interpolation, a technique that combines multiple translation models. Specifically, the DBMS-KU system interpolates a direct translation model (Baseline) with a pivot-based translation model. Russian is used as the pivot language, meaning translations are performed in two steps: Kazakh to Russian, then Russian to English (and vice-versa).

The authors explore two key aspects:

  • Pivot Language Strategy: This is employed to overcome the scarcity of direct Kazakh-English parallel data. By using Russian as an intermediary, they can leverage larger Russian-English and Kazakh-Russian corpora.
  • Language Model Order: They investigate the impact of using 3-gram and 5-gram language models. Language models are crucial for predicting the fluency and naturalness of the translated text. The paper hypothesizes that a higher n-gram order might lead to better results.

The phrase tables for the pivot system are constructed by merging source-to-pivot (SRC-PVT) and pivot-to-target (PVT-TRG) phrase tables, followed by pruning to reduce noise. The final interpolation combines the direct translation model with the pivot phrase table.

Datasets

The primary datasets used are from the WMT19 shared task for the Kazakh-English language pair. For the pivot system, they also utilize the Russian-English dataset from WMT18. The paper provides statistics for training and development sets for both the baseline and interpolation systems, noting the significant difference in size between the Kazakh-English and English-Kazakh training corpora.

Key Insights and Ideas

  • Pivot Power for Low-Resource Pairs: The interpolation system, by incorporating a pivot language (Russian), demonstrably outperforms the direct translation baseline for Kazakh-English. This highlights the effectiveness of pivot strategies when direct parallel data is scarce.
  • LM Order Matters, Especially for Kazakh-English: The 5-gram language model order showed a more significant positive impact on the BLEU-cased score for Kazakh-English translation compared to the 3-gram order, in both baseline and interpolation systems. This suggests that capturing longer-range dependencies in the language model is beneficial for this language pair.
  • Perplexity as a Quality Indicator: The paper uses perplexity to evaluate the quality of language models. Lower perplexity generally indicates a better model. Interestingly, for English-Kazakh, the interpolation system with a 3-gram LM achieved a lower perplexity than the 5-gram LM, a finding the authors note warrants further investigation.
  • Data Size Influence: The significantly larger training corpus for Kazakh-English (compared to English-Kazakh) correlated with higher BLEU scores, suggesting that the amount of target language data plays a crucial role in translation performance.

Key Results

  • The interpolation system achieved better BLEU-cased scores than the baseline system for both Kazakh-English (KK-EN) and English-Kazakh (EN-KK) translation directions.
  • Improvements in BLEU-cased score from baseline to interpolation were +0.1 and +0.5 points for KK-EN (using 3-gram and 5-gram LMs, respectively) and +0.1 points for EN-KK (for both LM orders).
  • The 5-gram LM order led to improvements of +0.3 and +0.7 points for KK-EN in the baseline and interpolation systems, respectively.
  • For EN-KK, the language model order did not significantly improve the BLEU-cased score.
  • The lowest perplexity for KK-EN was achieved by the 5-gram baseline system (45.51).
  • The lowest perplexity for EN-KK was achieved by the 5-gram baseline system (77.18).

Limitations and Future Work

  • Standardized Tokenizer: The authors acknowledge that using a standardized tokenizer from Moses might not be optimal for Kazakh and Russian. Future work could involve developing specific tokenizers for these languages to improve segmentation and, consequently, BLEU scores.
  • Pivot Language Choice: Exploring other pivot languages, particularly those with similar language families or word orders to Kazakh, could yield further improvements.
  • Interpolation Schemes: Investigating different interpolation schemes and applying them to Neural Machine Translation (NMT) systems, especially with out-domain datasets, is suggested to address the scarcity of Kazakh resources.
  • Morph-Based Language Modeling: The utilization of morph-based language modeling is also proposed as a future research direction.

Summary updated

BibTeX
@inproceedings{budiwati2019dbms,
  author = {Sari Dewi Budiwati and Al Hafiz Akbar Maulana Siagian and Tirana Noor Fatyanosa and Masayoshi Aritsugi},
  title = {{DBMS-KU Interpolation for WMT19 News Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure