Tilde’s Machine Translation Systems for WMT 2019

Marcis Pinnis, Rihards Krišlauks, Matīss RiktersACL 2019

TL;DR

Tilde's team developed neural machine translation (NMT) systems for the WMT 2019 news translation task, achieving a shared first place in human evaluation. They combined established Transformer architectures with novel filtering techniques and optimization methods, demonstrating strong performance even with limited computational resources.

What Problem Does This Paper Address?

The paper addresses the ongoing challenge of improving neural machine translation (NMT) quality, particularly for the English-Lithuanian and Lithuanian-English language pairs in the context of the WMT 2019 shared task. While Transformer models have become dominant, researchers are constantly seeking ways to enhance their performance through better data handling, optimization, and domain adaptation, especially under resource constraints. The authors aim to leverage recent advancements from 2018 to build production-ready NMT systems.

Methods

The core of Tilde's approach involves building upon the Transformer architecture and applying a series of refined techniques:

  • Data Filtering: They combined Tilde's MT filtering (TMTF) with Dual Conditional Cross-Entropy Filtering (DCCEF) to address issues like misalignment, incomplete translations, and data corruption. This combined approach proved more effective for unconstrained systems.
  • Data Pre-processing: Standard pre-processing workflows were maintained, including supplementing training corpora with synthetic data by replacing words with unknown tokens to improve handling of rare phenomena.
  • Optimizer: They experimented with the Quasi-Hyperbolic Adam (QHAdam) optimizer and a modified version (~QHAdam), finding it performed well, especially with specific learning rate and warm-up step configurations.
  • Large Batches: They confirmed that using large batch sizes, coupled with increased learning rates, leads to better-performing NMT systems, a finding consistent with prior research.
  • Back-translation: This was a key technique for domain adaptation. They explored various strategies for selecting data for back-translation, including targeting rare words and using language models to score sentences. They also compared beam search versus sampling-based decoding for generating back-translated data, finding sampling to be generally superior.
  • Source Domain Adherence: A novel method was introduced to ensure that back-translated data better reflects the source domain, addressing potential mismatches between source and target language writing styles.
  • Transformer-Big Configuration: For unconstrained systems, they explored a larger Transformer variant, which showed promise despite time constraints limiting full experimentation.
  • Automatic Named Entity Post-Editing (ANEPE): They extended their existing ANEPE workflow to handle multi-word named entities and non-translatable phrases, improving translation quality by fixing these specific translation errors.

Datasets

  • Training Data:
    • Constrained and unconstrained parallel corpora provided by the WMT 2019 organizers.
  • Proprietary news corpora from the Tilde Data Library.
  • Monolingual news data for language model training and back-translation.
  • Synthetic data generated by replacing words with unknown tokens.
  • Validation Data: The first 1000 sentences of the NewsDev2019 dataset.
  • Evaluation Data: NewsTest 2019 dataset.

Key Insights and Ideas

  • Resourcefulness Pays Off: The team demonstrated that significant improvements in NMT quality can be achieved even with limited computational resources (one or two GPUs) by strategically combining existing and novel techniques.
  • Filtering Synergy: Combining TMTF and DCCEF filtering proved beneficial, particularly for unconstrained systems, highlighting the importance of robust data cleaning.
  • Back-translation Nuances: The method of selecting data for back-translation matters. While targeting rare words didn't yield expected results, using language models to score and select sentences showed promise for domain adherence. Sampling-based decoding for back-translation was generally more effective than beam search.
  • Domain Adherence is Crucial: Ensuring that back-translated data truly reflects the source domain is vital for adapting NMT systems to specific content types, leading to better translation quality.
  • ANEPE Extension: Expanding the automatic named entity post-editing to handle multi-word entities and phrases significantly boosts translation quality by addressing common errors.

Key Results

  • Tilde's systems achieved a shared first place in human evaluation for the WMT 2019 news translation task.
  • The combined filtering approach (TMTF + DCCEF) showed improved performance for unconstrained systems.
  • The ~QHAdam optimizer, with specific settings (learning rate 5x10⁻⁴, 48k warm-up steps), performed well.
  • Large batch training, combined with increased learning rates, yielded better results.
  • Sampling-based decoding for back-translation generally outperformed beam search.
  • The ANEPE workflow improved translation quality by 0.04 to 0.1 BLEU points for all submissions.
  • Specific systems, like the '(u) so-beam-~qh-1-to-1' for Lt → En, showed strong source domain adherence and surpassed ensemble models.

Limitations and Future Work

  • Resource Constraints: The authors acknowledge that time and computational resource limitations prevented them from exhaustively testing all combinations of hyper-parameters and methods. This led to some experiments being performed on only one translation direction or with a limited number of configurations.
  • Iterative Approach: The iterative nature of their experiments meant that sometimes a new optimizer was adopted along with a new learning rate schedule without confirming that the baseline optimizer wouldn't also benefit from the schedule. This makes it difficult to definitively attribute improvements to a single change.
  • Transformer-Big Experimentation: The Transformer-big configuration was explored but not fully optimized or tested due to time constraints, suggesting potential for further gains.
  • Evaluation Strategy Change: The authors note that a change in the evaluation strategy for NewsTest 2019 (using texts originally written in the source language) might have favored systems with stronger source domain adherence, a factor they did not fully account for when selecting submission models.
  • Future Work: The paper implicitly suggests that further exploration of domain adaptation techniques, optimization strategies, and larger model architectures could lead to even greater improvements. The success of ANEPE also points towards continued development in automatic post-editing.

Summary updated

BibTeX
@inproceedings{pinnis2019tildes,
  author = {Marcis Pinnis and Rihards Krišlauks and Matīss Rikters},
  title = {{Tilde’s Machine Translation Systems for WMT 2019}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure