Tilde’s Machine Translation Systems for WMT 2018

Mārcis Pinnis, Matīss Rikters, Rihards KrišlauksEMNLP 2018

TL;DR

Tilde's WMT 2018 submission showcases the superiority of Transformer models over older RNN-based architectures for machine translation, particularly for Estonian-English. They also demonstrate that augmenting training data with back-translation, even double back-translation, significantly boosts performance, especially for constrained translation tasks.

What Problem Does This Paper Address?

The paper tackles the ongoing challenge of improving machine translation (MT) quality, specifically for the Estonian-English language pair, within the context of the WMT 2018 shared task. The field of Neural Machine Translation (NMT) has seen rapid architectural evolution, with a shift from Recurrent Neural Networks (RNNs) to Transformer models. This paper investigates how these newer architectures perform and how data augmentation techniques, like back-translation, can further enhance translation quality, especially for morphologically rich and less-resourced languages like Estonian. They also explore system ensembling and averaging as methods to achieve state-of-the-art results.

Methods

The core of Tilde's approach relies on the Transformer architecture, which has become the de facto standard for NMT. They employed two main data preparation workflows: a "full workflow" for constrained and unconstrained systems, and a "light workflow" for a specific system.

Key methodological aspects include:

  • Data Filtering and Pre-processing: Rigorous filtering of parallel corpora to remove noise and corruption was crucial. This involved techniques like content overlap filtering and language adherence checks. Pre-processing included HTML tag removal, normalization, and replacing non-translatable entities with placeholders.
  • Subword Units: Byte-Pair Encoding (BPE) was used to split tokens into subword units, a standard practice for handling rare words and morphological variations.
  • Synthetic Data Generation:
    • Back-translation: Monolingual data was translated back into the source language using existing MT systems to create synthetic parallel data. They explored using both initial Transformer models and pre-existing MLSTM models for this.
  • Unknown Token Infusion: To improve robustness to phenomena like code-mixing or rare words, they replaced aligned content words with unknown token identifiers.
  • Factored Data: For constrained systems, they utilized factored data, incorporating morphological and syntactic information as separate features.
  • Model Training: They used standard NMT toolkits like Nematus (for MLSTM) and Sockeye (for Transformer models), training until convergence.
  • Ensembling and Averaging: To boost performance, they employed model averaging (averaging weights of multiple trained models) and ensembling (combining outputs of different models, often through majority voting).

Datasets

The primary datasets used were the parallel corpora provided by the WMT 2018 organizers for news translation. Additionally, Tilde leveraged their Tilde Data Library, which contains publicly available and proprietary corpora.

They also generated significant amounts of synthetic data through back-translation and unknown token infusion, effectively doubling the size of their training data in some cases.

Key Insights and Ideas

  • Transformer Dominance: The paper strongly reinforces the trend that Transformer models significantly outperform older MLSTM-based RNN architectures for NMT, even for languages with complex morphology like Estonian.
  • The Power of Back-Translation: Double back-translation proved to be a highly effective strategy for improving translation quality, particularly for constrained translation tasks. This highlights the value of augmenting limited parallel data with synthetic data.
  • Data Quality Over Quantity (to an extent): While larger datasets are generally beneficial, the paper notes that the difference between constrained (smaller, cleaner) and unconstrained (much larger, potentially noisier) datasets was "rather minimal," suggesting that current NMT architectures might not fully exploit massive datasets without careful curation.
  • Ensembling for Top Performance: For constrained scenarios, ensembling averaged models yielded the best results, demonstrating that combining multiple well-trained models is a robust strategy for achieving peak performance.
  • Named Entity Handling: The paper describes a post-editing approach for named entities, using dictionaries and similarity metrics to correct translations, showing a marginal but positive impact on quality.

Key Results

  • Tilde submitted seven NMT systems for the WMT 2018 shared task, covering both English-Estonian and Estonian-English directions, in both constrained and unconstrained settings.
  • Their unconstrained systems (tilde-nc-nmt) achieved the best results among all submissions for both translation directions.
  • Their constrained systems were among the second highest-ranked for constrained scenarios.
  • Automatic evaluation (SacreBLEU) showed Transformer models consistently outperforming MLSTM models.
  • The paper presents detailed tables of automatic evaluation results (Table 4) and highlights the top-performing systems from the official human evaluation (Table 5), where Tilde's systems ranked highly.

Limitations and Future Work

  • Data Sparsity: While back-translation helps, the paper implicitly acknowledges that data scarcity, especially for morphologically rich languages, remains a challenge.
  • Exploiting Large Datasets: The observation that the performance difference between constrained and unconstrained systems was "rather minimal" despite a 17x data size difference suggests that future work could focus on better methods for learning from extremely large, potentially noisy, datasets.
  • Named Entity Post-editing Impact: The automatic post-editing for named entities had a "marginal impact," indicating room for improvement in this area.
  • System Combination Limitations: The combination for English-Estonian was not performed due to a lack of support for alignment extraction in Meteor for that language pair, suggesting a need for more robust cross-lingual alignment tools.
  • Future Architectures: The authors hint that the state-of-the-art architectures will continue to evolve beyond Transformers, citing models like RNMT+ that combine LSTMs with attention.

Summary updated

BibTeX
@inproceedings{pinnis2018tildes,
  author = {Mārcis Pinnis and Matīss Rikters and Rihards Krišlauks},
  title = {{Tilde’s Machine Translation Systems for WMT 2018}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure