Machine Translation with parfda, Moses, kenlm, nplm, and PRO

Ergun BiçiciACL 2019

TL;DR

This paper introduces parfda, a novel instance selection method that significantly improves statistical machine translation (SMT) systems by intelligently selecting training data. By integrating neural language models into the established Moses SMT framework, the authors demonstrate competitive translation performance and introduce a new test suite for evaluating MT systems on out-of-domain data.

What Problem Does This Paper Address?

Building high-performing statistical machine translation (SMT) systems traditionally relies on vast amounts of parallel data. However, as datasets grow, efficiently selecting the most relevant training instances becomes crucial. Furthermore, while neural machine translation (NMT) has gained prominence, SMT systems, especially when enhanced with advanced language modeling techniques, can still offer competitive performance and valuable insights. This paper tackles the challenge of optimizing SMT system development by proposing a smarter data selection strategy and exploring the integration of modern neural language models within a classic SMT pipeline. The authors also aim to provide a more realistic evaluation of MT systems by introducing an out-of-domain test suite.

Methods

The core innovation presented is parfda (parallel feature decay algorithms), an instance selection algorithm designed to intelligently select training and language model instances for SMT systems. parfda works by decaying feature weights, allowing for faster and more accurate SMT system construction. The authors integrate this with the well-known Moses SMT toolkit.

Key methodological aspects include:

  • Instance Selection: parfda uses the test set's source sentences to select training data and the corresponding target side for language model data. It decays weights for both source and target features to enhance diversity.
  • Hybrid Language Models: The paper explores integrating neural language models (NPLM) with traditional n-gram based language models (kenlm) within the Moses framework. This hybrid approach aims to leverage the strengths of both.
  • Truecasing: A truecasing strategy is applied before and after decoding to improve translation quality.
  • Optimization: Pairwise ranking optimization (PRO) is used for tuning parameters, achieving robust results with fewer iterations.

Datasets

The research utilizes existing datasets from the WMT19 news translation task for training and evaluation. Additionally, the authors introduce a new test suite specifically designed for the German-English language pair. This test suite is intentionally out-of-domain, focusing on sentences related to international relations and cultural artifacts, sourced from press releases of the Prussian Cultural Heritage Foundation. This aims to provide a more challenging and realistic evaluation scenario compared to in-domain news data.

Key Insights and Ideas

  • parfda's Effectiveness: The instance selection method parfda proves effective in building SMT systems, especially as parallel and monolingual data sizes increase. It allows for the creation of SMT systems that can achieve competitive results.
  • Neural LM Integration: Combining n-gram LMs (kenlm) with neural LMs (nplm) within the Moses SMT framework yields improvements in translation performance.
  • Hyphenated Words Matter: Splitting hyphenated words significantly boosts translation quality, suggesting that finer-grained tokenization can be beneficial.
  • Out-of-Domain Evaluation is Crucial: The new German-English test suite highlights the limitations of evaluating MT systems solely on in-domain data, demonstrating that performance can degrade significantly on unseen domains.
  • Upper Bounds Provide Context: Analyzing translation upper bounds based on target coverage (tcov) helps in understanding the difficulty of different translation directions and provides a benchmark for system performance.

Key Results

The paper reports several key quantitative and qualitative findings:

  • Competitive SMT Performance: The parfda-enhanced Moses SMT systems, particularly when combined with n-gram and neural LMs, achieve competitive results on various language pairs in the WMT19 task. For instance, Table 4 shows parfda results are comparable to top constrained submissions, with an average BLEU difference of 14.05 points in 2019.
  • Data Statistics: Table 1 provides detailed statistics on training and LM corpora size and coverage for different language pairs, comparing the parfda selected data with the constrained (C) setting.
  • Testsuite Performance: Table 10 shows the F₁ scores for the German-English test suite, with parfda achieving respectable scores, though generally lower than top-performing systems on in-domain tasks.
  • Language Model Gains: Table 3 indicates that using nplm alone can decrease scores, but improvements are seen when nplm is used in conjunction with kenlm.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Computational Cost of NPLM: Building NPLM models is computationally intensive, taking significantly longer and requiring more disk space than kenlm.
  • Testsuite Evaluation: While the new German-English test suite provides valuable out-of-domain evaluation, the authors plan to extend this work towards more objective key phrase evaluation methods.
  • Upper Bound Optimism: The calculated BLEU upper bounds are optimistic as they do not account for reordering or sentence length differences.
  • Future Work: The authors aim to extend their work to more objective key phrase evaluation methods and explore further improvements in instance selection and language model integration.

Summary updated

BibTeX
@inproceedings{bicici2019machine,
  author = {Ergun Biçici},
  title = {{Machine Translation with parfda, Moses, kenlm, nplm, and PRO}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure