PROMT Systems for WMT 2019 Shared Translation Task

Alexander MolchanovACL 2019

TL;DR

PROMT's WMT 2019 submission showcases their neural machine translation systems for English-Russian, English-German, and German-English. They highlight improvements in data filtering and a novel approach to handling named entities using a special <UNK> token, achieving competitive results in two out of three language pairs.

What Problem Does This Paper Address?

The paper addresses the ongoing challenge of building high-quality neural machine translation (NMT) systems, particularly for the WMT Shared Translation Task. This involves optimizing data preparation, model architecture, and training strategies to achieve state-of-the-art performance. Specifically, the authors focus on improving translation quality by refining data filtering techniques and developing robust methods for handling named entities, which are often problematic for NMT models. They also aim to build general-domain models, not just news-specific ones, by leveraging diverse datasets.

Methods

The PROMT team utilized the MarianNMT toolkit for their NMT systems. Key methodological aspects include:

  • Data Filtering Pipeline: A multi-stage process involving basic length-based heuristics, removal of tags, and filtering of lines with low alphabetic content. A significant improvement was made by using a modified bilingual Moore-Lewis data selection algorithm, incorporating language models to filter noisy data and remove similar sentence pairs.
  • Byte Pair Encoding (BPE): They employed BPE for subword tokenization. A notable innovation is their "trick" to handle casing by inserting special tokens (<C>, <U>) for title case or uppercase sequences, mimicking case-insensitive BPE without direct support in their chosen toolkit.
  • Named Entity (NE) Handling: A novel approach involves producing a baseline NMT translation, validating NEs, and if errors are found, replacing problematic NEs with a special <UNK> placeholder in the source sentence for re-translation. The <UNK> placeholders are then substituted back with their original values. This method aims to preserve NEs while allowing the model to learn their context.
  • Ensemble and Fine-tuning: For English-Russian, they used an ensemble of two fine-tuned models. For English-German, a single fine-tuned model was used, and for German-English, a single baseline model.
  • Rule-Based Machine Translation (RBMT) Fallback: The system falls back to RBMT in specific cases, such as when the NMT output language is incorrect, the output contains recurring words/n-grams, or the output length is significantly different from the input.

Datasets

The authors used a combination of datasets:

  • WMT Organizers' Data: Provided parallel data for the shared task.
  • Private In-house Parallel Data: Proprietary data used for training.
  • Publicly Available Data: Primarily from the OPUS website (e.g., Europarl, DGT, EMEA, ECB, NewsCommentary, TED2013, GlobalVoices).
  • Tatoeba Corpus: Used as validation sets, chosen for its general-domain nature and availability across many language pairs.
  • News 2018 Corpora (statmt.org): Used for fine-tuning and as in-domain data for data selection.
  • Wikipedia Dumps: Used for back-translation and data selection.
  • OpenSubtitles: A large corpus used, but with careful filtering due to its noisy nature.
  • Newstest2018: Used as the test set.

Key Insights and Ideas

  • Data Filtering is Crucial: The paper strongly suggests that improved data filtering, especially using language models and identifying similar sentence pairs, significantly boosts translation quality.
  • <UNK> for NEs: The <UNK> placeholder strategy for handling named entities is a clever workaround to preserve them while allowing the NMT model to learn their context, proving more effective than previous methods.
  • General-Domain vs. News-Domain: Using the Tatoeba corpus for validation signals a move towards building more versatile NMT models that can perform well beyond just the news domain.
  • BPE Casing Trick: The method of inserting special tokens for casing is an effective way to inject case information into BPE-based subword units when the architecture doesn't natively support it.
  • Smaller BPE Models: They found that smaller BPE models (35k-45k merge operations) performed better for English-Russian, indicating that overly large vocabularies can lead to more out-of-vocabulary (OOV) issues.

Key Results

  • English-Russian: The fine-tuned ensemble of Model1 and Model2 achieved a BLEU score of 32.9 on newstest2018 and 30.8 on newstest2019, significantly outperforming their previous year's baseline (27.4 and 24.7 respectively).
  • English-German: The fine-tuned model achieved 40.4 on newstest2018 and 38.4 on newstest2019.
  • German-English: The baseline model achieved 40.1 on newstest2018 and 32.1 on newstest2019. The authors note this pair's lower quality, attributing it to the data being primarily filtered for the English-German direction.
  • Fine-tuning Impact: Fine-tuning provided only marginal improvements in BLEU scores, suggesting a need for alternative data selection approaches for domain adaptation.

Limitations and Future Work

  • German-English Data Bias: The authors acknowledge that the German-English model's performance was likely impacted because the training data was primarily filtered with the English-German direction in mind, leading to less attention on the English side.
  • RBMT Fallback for English-Russian: The English-Russian model occasionally generated English text, necessitating a fallback to RBMT. This was traced back to English sentences in the Russian news corpus used for back-translation.
  • Future Work:
    • Improving data filtering, especially for bidirectional training.
  • Developing better methods for handling proper name translation into Russian.
  • Exploring other language pairs, including Chinese and Kazakh.
  • Investigating new approaches for data selection for domain adaptation.

Summary updated

BibTeX
@inproceedings{molchanov2019promt,
  author = {Alexander Molchanov},
  title = {{PROMT Systems for WMT 2019 Shared Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure