PROMT Systems for WMT 2018 Shared Translation Task

Alexander MolchanovEMNLP 2018

TL;DR

PROMT's WMT 2018 submission showcases two primary neural machine translation (NMT) systems for English-Russian, one based on Marian and a hybrid NMT system incorporating OpenNMT for post-editing. Both systems significantly outperformed their rule-based baseline, demonstrating the effectiveness of modern NMT architectures even in challenging, low-resource scenarios.

What Problem Does This Paper Address?

The paper tackles the challenge of building high-quality neural machine translation (NMT) systems for the English-Russian language pair, a task that often presents difficulties due to linguistic differences and the availability of parallel data. Specifically, the authors aim to:

  • Develop competitive NMT systems: The WMT Shared Task provides a benchmark for evaluating translation quality, and the goal is to achieve state-of-the-art performance.
  • Explore hybrid approaches: The paper investigates combining the strengths of NMT with existing rule-based machine translation (RBMT) systems, particularly for handling specific linguistic phenomena.
  • Address data scarcity: While large parallel corpora exist, domain-specific or high-quality data for certain language pairs can be limited. The authors explore techniques to leverage available data effectively.
  • Improve translation of proper names: Proper names are notoriously difficult for NMT systems, often leading to mistranslations or incorrect casing. The paper aims to mitigate this issue.

Methods

PROMT submitted three systems, with two primary NMT-based approaches:

  • Pure NMT System (Marian): This system utilizes the Marian NMT toolkit, a fast and efficient framework for neural machine translation. It employs a standard transformer architecture. Key aspects include:
  • Byte Pair Encoding (BPE): Used for subword tokenization with 85K operations for both source and target languages. A shared vocabulary was not used due to the distinct Cyrillic alphabet of Russian.
  • Data Augmentation: The system was trained on a combination of provided WMT data, private in-house news data, TED Talks corpus, and significantly, synthetic data generated through back-translation and a novel "replicated data with unknown words" technique.
  • Proper Name Handling: A sophisticated pipeline was developed to identify proper names using Stanford NER, check their frequency and BPE splitting in the training data, replace them with <UNK> placeholders, translate using a model trained to handle unknown words, and finally substitute the placeholders with RBMT translations of the names. This system was designed to reproduce unknown words.
  • Back-off to RBMT: For a small percentage of cases (less than 1% of validation sentences), the system falls back to the RBMT system if language recognition fails or if recurring words/n-grams are detected, which are known NMT issues.
  • Hybrid NMT System (OpenNMT + RBMT): This system combines a neural post-editing component with PROMT's existing RBMT engine.
  • OpenNMT Post-Editing: A deep bidirectional encoder-decoder model with attention (based on Luong et al., 2015) was trained using OpenNMT. This model acts as a post-editor for the RBMT output.
  • Data: Trained on initial parallel data, back-translated data, and augmented with monolingual data.
  • No Special Name Handling: Unlike the pure NMT system, this hybrid system did not implement special name processing or fall back to RBMT for names, as the post-editing component was deemed sufficient.
  • Pure RBMT System: A baseline system using PROMT's mature rule-based machine translation engine.

Datasets

The following datasets were used for training and evaluation:

  • Training Data:
    • WMT organizers' provided data.
  • Private in-house news parallel data (approx. 600k sentences).
  • TED Talks corpus (from OPUS).
  • NewsCommentary corpus.
  • CommonCrawl corpus (heavily filtered).
  • ParaCrawl corpus (heavily filtered).
  • MultiUN corpus (filtered for sentence length ratio).
  • Yandex corpus.
  • Monolingual Russian news corpus (2017) for the hybrid system.
  • Synthetic Data: * Back-translated data: Generated by training auxiliary NMT systems (Russian-English Marian and Russian-RBMT OpenNMT) and using them to translate monolingual Russian data.
  • Replicated data with unknown words: Created by word-aligning parallel sentences and randomly replacing aligned subword units with <UNK> placeholders.
  • Validation Set: WMT newstest2017.
  • Test Sets: WMT newstest2017 and newstest2018.

Key Insights and Ideas

  • NMT Dominance: The pure NMT system, particularly the transformer-based Marian system, emerged as the strongest performer, significantly outperforming the RBMT baseline. This reinforces the trend of NMT systems achieving superior translation quality.
  • Hybrid System Effectiveness: The hybrid NMT system, while not the top performer, still achieved competitive results, suggesting that combining NMT post-editing with RBMT can be a viable strategy, especially when RBMT has strong linguistic features.
  • Impact of Proper Name Handling: The dedicated proper name processing pipeline for the pure NMT system, while slightly reducing BLEU score, was chosen for submission due to its potential for improved accuracy in translating names. This highlights the trade-off between overall fluency and specific linguistic accuracy.
  • Data Augmentation is Crucial: The paper emphasizes the significant role of synthetic data generation (back-translation and unknown word replication) in boosting the performance of NMT systems, especially in scenarios with limited high-quality parallel data.
  • Challenges with Inflectional Languages: The authors note that Russian's highly inflective nature poses challenges for NMT, particularly in correctly casing translated names when using subword units and placeholders.

Key Results

  • BLEU Scores:
    • RBMT: 22.9 (newstest2017), 18.1 (newstest2018)
    • Pure NMT (Marian): 31.0 (newstest2017), 27.4 (newstest2018)
    • NMT+names (pure NMT with name processing): 30.9 (newstest2017), 27.3 (newstest2018)
    • Hybrid NMT (OpenNMT post-editing): 29.5 (newstest2017), 25.3 (newstest2018)
  • The pure NMT system achieved the highest BLEU scores on both test sets.
  • The NMT+names system's BLEU score was slightly lower than the default pure NMT system, but the authors prioritized its name translation accuracy.
  • The proper name processing pipeline affected 27% of sentences in the test set.

Limitations and Future Work

  • Proper Name Casing: A significant limitation noted is the inability of the Marian system to output correct casing for translated names when using placeholders. This is attributed to the lack of factored translation support in Marian.
  • Subword Unit Training: The system was trained to reproduce placeholders for subword units, not whole words, which impacted the generation of synthetic data for name translation.
  • Future Work: The authors suggest improving the proper name translation pipeline by teaching the system to output translations in the correct case. Further research could explore more advanced techniques for handling rare words and named entities in morphologically rich languages.

Summary updated

BibTeX
@inproceedings{molchanov2018promt,
  author = {Alexander Molchanov},
  title = {{PROMT Systems for WMT 2018 Shared Translation Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure