Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data

Marcin Junczys-DowmuntEMNLP 2018

TL;DR

Microsoft's WMT2018 English-German translation system achieved top scores by combining state-of-the-art Transformer models with novel data filtering and weighting techniques. Their "dual conditional cross-entropy filtering" method significantly improved translation quality by better identifying and utilizing high-quality parallel data.

What Problem Does This Paper Address?

The paper addresses the challenge of building high-performing machine translation systems, specifically for the WMT2018 English-German news translation task. While Transformer models have become the de facto standard, the authors highlight that the quality and quantity of training data, especially for noisy, large-scale parallel corpora like Paracrawl, remain critical bottlenecks. Effectively filtering and leveraging this data is crucial for pushing the boundaries of translation accuracy, especially when aiming for top performance in competitive shared tasks.

Methods

The core of Microsoft's submission lies in their data processing pipeline. They utilized the efficient Marian NMT toolkit and employed standard Transformer-big architectures. The novelty comes from their data enhancement strategies:

  • Dual Conditional Cross-Entropy Filtering: This is the paper's main contribution. It involves training two translation models in inverse directions (English-to-German and German-to-English) on a clean, in-domain dataset. A score is then calculated for each sentence pair in a noisy corpus (like Paracrawl) based on the agreement between these two models' conditional probabilities. This score helps identify sentence pairs that are likely to be good translations in both directions.
  • Sentence Weighting: Building on the filtering score, they also experimented with weighting individual sentence pairs during training. Sentences with higher scores contribute more to the training process, effectively prioritizing high-quality data.
  • Standard Techniques: They also incorporated established techniques like back-translation (generating synthetic parallel data) and ensembling with a Transformer-style language model.

Datasets

  • WMT2018 News Translation Task Data: This was the primary dataset for training and evaluation.
  • Paracrawl: A large, noisy parallel corpus used for experimentation with data filtering and weighting.
  • WMT German News Data (2015-2017): Used as a clean, in-domain dataset for training the filtering models.
  • WMT2016/2017 test sets: Used for evaluation during development.

Key Insights and Ideas

  • Data is King (Still): Despite the power of Transformer models, the paper strongly emphasizes that sophisticated data filtering and weighting can yield significant improvements, even outperforming architectural tweaks or more complex models.
  • Bidirectional Agreement is Key: The "dual conditional cross-entropy" filtering method is effective because it leverages the intuition that good translations should be probable in both directions. This is a more robust signal than relying on a single direction or language model.
  • Noisy Data Can Be Useful (with care): Large, noisy datasets like Paracrawl are valuable resources, but their utility hinges on effective filtering. Simply adding them without proper cleaning can degrade performance.
  • Weighting Matters: Instance weighting during training can further refine the impact of filtered data, allowing the model to focus on the most reliable examples.

Key Results

  • Automatic Metrics (BLEU): Microsoft's submission achieved the highest BLEU score for English-German translation, with a margin of nearly 2 BLEU points over the second-best system.
  • Human Evaluation: They ranked first among constrained systems based on human evaluation.
  • Data Filtering Impact: Selecting 8 million sentences from Paracrawl using their filtering method led to a gain of 1.4 BLEU points on the WMT2016 test data compared to using the full 32 million sentences without filtering. Weighting further improved results slightly.

Limitations and Future Work

  • Post-Submission Evaluation: Some of the reported results, particularly for the Transformer-big models with weighting, were obtained after the official submission deadline.
  • R2L Reranking: While effective for RNNs, right-to-left reranking showed less impact on Transformer models, possibly due to insufficient convergence time for those specific experiments.
  • Future Work: The authors suggest that further exploration of data weighting and filtering techniques, especially for eliminating harmful effects from bad data, is warranted. The effectiveness of their filtering method in other tasks (like the Parallel Corpus Filtering Task) also points to broader applicability.

Summary updated

BibTeX
@inproceedings{junczysdowmunt2018microsofts,
  author = {Marcin Junczys-Dowmunt},
  title = {{Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure