Panlingua-KMI MT System for Similar Language Translation Task at WMT 2019

Atul Kr. Ojha, Ritesh Kumar, Akanksha Bansal, Priya RaniACL 2019

TL;DR

This paper explores the effectiveness of both phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for closely related languages (Nepali-Hindi and Hindi-Nepali) in the WMT 2019 shared task. They found that PBSMT outperformed NMT, even with limited data, suggesting that the inherent similarities between these languages might not be as advantageous for NMT as initially expected, and that traditional PBSMT methods remain competitive.

What Problem Does This Paper Address?

Developing high-quality machine translation (MT) systems for languages that are closely related, like those within the same family, presents a unique challenge. While linguistic similarity might intuitively suggest an advantage (e.g., requiring less data), the authors aim to understand how to best leverage this similarity. Specifically, they investigate whether modern neural machine translation (NMT) systems, which are typically data-hungry, can outperform traditional phrase-based statistical machine translation (PBSMT) systems when dealing with low-resource, closely related language pairs, and if so, under what conditions.

Methods

The Panlingua-KMI team developed a total of 11 MT systems for the WMT 2019 Similar Language Translation Task, focusing on Nepali-Hindi and Hindi-Nepali. Their approach involved experimenting with two main paradigms:

  • Phrase-Based Statistical Machine Translation (PBSMT): They utilized the Moses toolkit, employing KenLM for language modeling and the 'grow-diag-final-and-heuristic' method for phrase extraction. Various pre-processing strategies were tested to understand their impact on performance.
  • Neural Machine Translation (NMT): They used the OpenNMT toolkit (PyTorch port) to build NMT systems. Two architectures were explored: a 2-layer LSTM model and a 6-layer Transformer model. Experiments were conducted with limited data and without leveraging monolingual data, to specifically test NMT's performance under these constraints.

Datasets

The experiments utilized parallel and monolingual datasets provided by the WMT 2019 Similar Translation Shared Task organizers.

  • Parallel Data: Sub-divided into training, tuning, and testing sets.
  • Nepali-Hindi: 65,505 sentences for training, 3,000 for tuning.
  • Hindi-Nepali: 92,296 sentences for training, 3,000 for tuning.
  • Monolingual Data:
    • Nepali: 104,967 sentences.
  • Hindi: Procured from the Workshop on Asian Translation Shared Task 2018.

Test sets consisted of 2,000 sentences for Nepali-Hindi and 1,567 sentences for Hindi-Nepali.

Key Insights and Ideas

  • PBSMT Outperforms NMT for Closely Related, Low-Resource Pairs: Surprisingly, PBSMT systems achieved better results than NMT systems in both BLEU score and Translation Error Rate (TER) for the tested language pairs. This challenges the common assumption that NMT universally surpasses PBSMT, especially in resource-constrained scenarios.
  • Language Similarity Doesn't Automatically Benefit NMT: The authors found that the inherent similarities between Nepali and Hindi did not provide a significant advantage for NMT systems. This suggests that simply having related languages doesn't guarantee better NMT performance without sufficient data and potentially more sophisticated adaptation techniques.
  • Pre-processing Matters for PBSMT: For PBSMT, different pre-processing strategies (like lowercasing and removing non-UTF characters) had a noticeable impact on performance, highlighting the importance of data cleaning for statistical methods.
  • Shorter Sentences are Better: Datasets with sentences up to 40 words performed better than those with sentences up to 80 words, indicating that sentence length can be a critical factor in translation quality for these language pairs.

Key Results

  • The Panlingua-KMI PBSMT system for Hindi-Nepali ranked 2nd out of 13 systems submitted for that pair.
  • The Nepali-Hindi PBSMT system ranked 4th out of 12 systems submitted for its pair.
  • For Hindi-Nepali, the best PBSMT system achieved a BLEU score of 11.5 and a TER of 79.1.
  • For Nepali-Hindi, the best PBSMT system achieved a BLEU score of 9.8 and a TER of 91.3.
  • Qualitative analysis (Figures 3 and 4) showed that PBSMT outputs were generally preferred over NMT outputs, with NMT systems sometimes failing to produce translations for certain source sentences due to issues like over-generation or out-of-vocabulary words.

Limitations and Future Work

  • NMT Data Hunger: The paper acknowledges that NMT systems are "extremely data-hungry" and require large amounts of parallel data to perform optimally. The experiments were specifically designed to test NMT under low-resource conditions, which inherently limited its potential.
  • Limited NMT Exploration: The NMT experiments were conducted with default hyper-parameters and without leveraging monolingual data, which could have potentially improved performance.
  • Future Research on Language Similarity: The authors suggest that to truly leverage the similarity between closely related languages, more sophisticated methods beyond standard NMT and PBSMT approaches need to be explored. This could involve techniques that explicitly model or exploit shared linguistic features.
  • Impact of Pre-processing: While pre-processing was found to be important for PBSMT, further research could explore its impact on NMT systems for these specific language pairs.

Summary updated

BibTeX
@inproceedings{ojha2019panlingua,
  author = {Atul Kr. Ojha and Ritesh Kumar and Akanksha Bansal and Priya Rani},
  title = {{Panlingua-KMI MT System for Similar Language Translation Task at WMT 2019}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure