CUNI Transformer Neural MT System for WMT18

Martin PopelEMNLP 2018

TL;DR

The CUNI Transformer system achieved state-of-the-art results in the WMT18 English-Czech and Czech-English news translation tasks by combining the Transformer architecture with several novel techniques. These include an improved iterative backtranslation process, a novel data filtering strategy, and a unique approach to tuning models for specific language domains.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems, particularly those based on the Transformer architecture, heavily rely on the quantity and quality of training data. While large parallel corpora are ideal, they are often scarce. This paper addresses the challenge of maximizing translation quality with limited authentic parallel data by leveraging monolingual data and improving existing techniques. Specifically, it tackles issues like the presence of noisy or irrelevant sentences in training data, the effective use of backtranslation for synthetic data generation, and the domain mismatch between training data and test sets.

Methods

The CUNI Transformer system builds upon the standard Transformer architecture (Vaswani et al., 2017). The authors introduce five key techniques to enhance its performance:

  • Improved Backtranslation (Concat Regime): Instead of mixing synthetic and authentic data or fine-tuning on synthetic data, they propose a "concat regime" where authentic and synthetic parallel data are simply concatenated. This approach, combined with checkpoint averaging, led to significant improvements. They also iterated this process, using the output of one translation direction to generate better synthetic data for the other.
  • Training Data Filtering: They identified and filtered out noisy sentences from the synthetic data. This included filtering out English sentences from Czech monolingual data (using an accented character heuristic) and removing sentences longer than 500 characters. They also filtered out sentences with repeated words or phrases to mitigate issues like filler words.
  • CZ/nonCZ Tuning: Recognizing that Transformer models perform differently on Czech-domain (CZ) and non-Czech-domain (nonCZ) test sets, they trained two separate models. One was tuned for CZ-domain articles, and the other for nonCZ-domain articles, by selecting checkpoints with the best performance on respective dev sets.
  • Coreference Pre-processing: To address the pro-drop nature of Czech, where pronouns can be omitted, they implemented a pre-processing step. This involved inserting the pronoun "ona" (she) in cases where coreference resolution indicated a female subject was dropped and the context pointed to a different sentence. This was a heuristic applied to improve translation accuracy for gender-ambiguous sentences.
  • RegEx Post-processing: Two simple post-processing steps were applied:
  • Removing repeated phrases (keeping only the first occurrence).
  • Converting English quotation marks to their Czech equivalents to adhere to formal writing conventions.

Datasets

  • Training Data: The system was trained on data allowed in the WMT18 shared task, including authentic parallel data (CzEng 1.7, Europarl v7, News Commentary v11, CommonCrawl) and synthetic data generated through backtranslation of monolingual data (Czech and English NewsCrawl articles).
  • Evaluation Data: The primary evaluation was performed on the WMT18 news translation test sets (newstest2017 and newstest2018) for English-Czech and Czech-English directions.

Key Insights and Ideas

  • Iterative Backtranslation Pays Off: The paper demonstrates that iteratively improving synthetic data quality by using the output of one translation direction to generate data for the other can lead to substantial gains.
  • Simple Filtering is Powerful: Basic filtering of synthetic data, especially removing noisy or irrelevant sentences, can significantly improve model performance by preventing the model from learning undesirable patterns.
  • Domain-Specific Tuning Matters: The observation that models trained on synthetic data perform better on CZ test sets and models trained on authentic data perform better on nonCZ test sets highlights the importance of domain adaptation, even within a single language pair.
  • Coreference Resolution as a Pre-processing Step: While NMT systems are typically sentence-level, incorporating context through coreference resolution for pronoun disambiguation can improve translation quality, especially for languages with pro-drop features.
  • The "Concat Regime" for Backtranslation: This approach, which avoids complex mixing or fine-tuning strategies, proved effective when combined with checkpoint averaging.

Key Results

The CUNI Transformer system significantly outperformed all other systems in the WMT2018 English-Czech and Czech-English news translation tasks.

  • On the English-Czech newstest2017, their final system achieved a BLEU score of 26.81 (cased), which was a substantial improvement over previous systems.
  • For the WMT2018 evaluation on newstest2018, their system achieved:
  • English-Czech: 26.01 (cased BLEU), 67.2 (refDA)
    • Czech-English: 33.91 (cased BLEU), 71.8 (refDA)
  • These results were significantly better (p < 0.05) than the second-best system (UEdin NMT) according to both automatic (BLEU bootstrap resampling) and manual (refDA Wilcoxon rank-sum) evaluations.

Limitations and Future Work

  • Manual Evaluation Reliance: The authors note that automatic metrics like BLEU can be misleading when MT quality is very high, as human references might not always be superior. They plan to conduct more extensive manual evaluations to assess the relative contribution of each technique.
  • Coreference Pre-processing as a Workaround: The coreference pre-processing is considered a temporary solution until document-level NMT becomes more readily available and integrated into frameworks like Tensor2Tensor.
  • Future Work: The authors intend to further assess the relative improvement of each of their five techniques through manual evaluation.

Summary updated

BibTeX
@inproceedings{popel2018cuni,
  author = {Martin Popel},
  title = {{CUNI Transformer Neural MT System for WMT18}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure