English-Myanmar NMT and SMT with Pre-ordering: NICT's Machine Translation Systems at WAT-2018

Rui Wang, Chenchen Ding, Masao Utiyama, Eiichiro SumitaWAT@PACLIC 2018 · OpenReview

TL;DR

This paper presents NICT's participation in the WAT-2018 shared task for English-Myanmar machine translation. They achieved top rankings using Transformer-based NMT and phrase-based SMT systems, both enhanced with pre-ordering techniques. The results highlight the effectiveness of modern NMT architectures and the subtle but beneficial impact of pre-ordering, especially for the SMT system.

What Problem Does This Paper Address?

Machine translation (MT) systems, particularly for low-resource languages like Myanmar (Burmese) and English, face challenges in achieving high-quality translations. Building effective MT systems requires robust architectures, sufficient parallel data, and often, sophisticated pre-processing or post-processing steps to handle linguistic differences. This paper addresses the challenge of developing state-of-the-art MT systems for the English-Myanmar language pair within the constraints of a shared task, where only provided parallel data can be used. Specifically, it explores the effectiveness of modern Neural Machine Translation (NMT) architectures, like the Transformer, and traditional Statistical Machine Translation (SMT) systems, and investigates the impact of pre-ordering on both.

Methods

The authors developed two main types of machine translation systems:

  • Neural Machine Translation (NMT): They employed the Transformer architecture, a popular and highly effective deep learning model for sequence-to-sequence tasks. For implementation, they used the Marian framework, known for its speed and support for advanced features. Their NMT systems were trained on the provided parallel data using the Transformer architecture with specific hyperparameters detailed in the paper (e.g., max-length 80, dec-depth 6, beam-size 12).
  • Statistical Machine Translation (SMT): They utilized the Moses toolkit for phrase-based SMT. This involved training word alignments and phrase tables using mgiza, symmetrizing alignments, and training 5-gram language models with SRILM. Model weights were tuned using MERT (Minimum Error Rate Training).
  • Pre-ordering: A key technique explored was pre-ordering. For English-to-Myanmar translation, they applied a dependency-based head finalization method to reorder the source English sentence before feeding it into both the NMT and SMT systems. This aims to align the word order of English more closely with that of Myanmar.
  • Ensemble Decoding: For NMT, they also employed ensemble decoding, combining the predictions of multiple independently trained models to improve robustness and performance.

Datasets

The primary datasets used were provided by the WAT-2018 organizers:

  • Training Data:
    • train(ALT): 17.9K lines.
  • train(UCSY): 208.6K lines.
  • Development/Test Data:
    • dev(ALT): 0.9K lines.
  • test(ALT): 1.0K lines.

The paper also mentions that the SMT systems used a 5-gram language model trained on the "entire target side of the parallel data."

Key Insights and Ideas

  • Transformer Dominance: The Transformer architecture, a standard in modern NMT, proved highly effective, achieving top rankings in the competition.
  • Pre-ordering's Nuance: While pre-ordering showed a slight improvement for the SMT system, its impact on the NMT system was not significant. This suggests that NMT architectures might be more adept at handling word order differences internally.
  • Ensemble Power: Ensemble decoding significantly boosted the performance of the NMT system, demonstrating the benefit of combining multiple models.
  • Language Pair Specifics: The performance differences between English-to-Myanmar and Myanmar-to-English highlight the inherent challenges and asymmetries in translating between these languages, especially concerning word order.

Key Results

The paper reports results using BLEU scores and official human evaluation on the test(ALT) set.

  • NMT Performance: The NICT's NMT system (Marian ensemble) achieved the first rank in English-to-Myanmar translation according to official human evaluation (61.00) and a strong BLEU score (29.89). For Myanmar-to-English, it ranked second with a BLEU score of 20.79.
  • SMT Performance: The SMT system (Moses) achieved a BLEU score of 9.44 for Myanmar-to-English and 25.55 for English-to-Myanmar.
  • Impact of Pre-ordering:
    • Pre-ordering improved the SMT system's Myanmar-to-English BLEU score by 0.2 points (from 9.44 to 9.64, though this specific number isn't explicitly stated, the text implies a small improvement).
  • Pre-ordering did not show significant improvement for the NMT systems.
  • Ensemble vs. Single Model: Ensemble decoding significantly outperformed decoding with a single NMT model.

Limitations and Future Work

  • Data Constraints: The systems were trained using only the provided parallel data, which is a common limitation in shared tasks but can restrict the potential for higher performance if more data were available.
  • Pre-ordering Effectiveness: The limited impact of pre-ordering on NMT suggests that further research might be needed to find more effective pre-ordering strategies for Transformer-based models or to explore alternative methods for handling word order differences.
  • Future Work: The paper doesn't explicitly detail future work, but the findings suggest exploring more advanced pre-ordering techniques or focusing on other linguistic phenomena that might be more impactful for English-Myanmar translation.

Summary updated

BibTeX
@inproceedings{wang2018english,
  author = {Rui Wang and Chenchen Ding and Masao Utiyama and Eiichiro Sumita},
  title = {{English-Myanmar NMT and SMT with Pre-ordering: NICT's Machine Translation Systems at WAT-2018}},
  booktitle = {WAT@PACLIC},
  year = {2018},
  url = {https://openreview.net/forum?id=cqoI6aOlD5},
}
Figure