The University of Maryland's Chinese-English Neural Machine Translation Systems at WMT18

Weijia Xu, Marine CarpuatWMT (shared Task) 2018 · OpenReview

TL;DR

The University of Maryland's WMT18 Chinese-English Neural Machine Translation (NMT) systems achieved state-of-the-art results by leveraging Transformer architectures, extensive data augmentation through backtranslation, and sophisticated ensembling and reranking techniques. This work demonstrates the continued dominance of Transformers and highlights the asymmetric impact of various improvements across translation directions.

What Problem Does This Paper Address?

Machine translation, particularly between low-resource languages like Chinese and English, remains a challenging task. While deep learning, specifically neural networks, has revolutionized NMT, achieving top performance often requires complex, resource-intensive models and training pipelines. This paper addresses the need for high-quality NMT systems that are inspired by state-of-the-art approaches but are more efficient in terms of training time and memory requirements. The authors aim to evaluate and improve upon existing Transformer-based models for the WMT18 Chinese-English news translation tasks.

Methods

The core of the University of Maryland's submission is the Transformer architecture, a powerful sequence-to-sequence model that relies on self-attention mechanisms. This allows it to capture long-range dependencies in text more effectively than traditional Recurrent Neural Networks (RNNs).

To boost performance, several key techniques were employed:

  • Byte-Pair Encoding (BPE): This subword tokenization method helps handle rare words and reduces the vocabulary size, improving generalization.
  • Backtranslation: Monolingual data, which is often abundant, is translated back into the source language using a preliminary NMT model. This synthetic parallel data is then used to augment the original training corpus, significantly improving translation quality.
  • Ensembling: Multiple Transformer models trained with different random seeds are combined to produce a more robust prediction.
  • Reranking: To further refine the translation output, the system utilizes n-best lists (multiple translation hypotheses) generated by the primary model. These hypotheses are then rescored using features from auxiliary models:
  • Right-to-Left (R2L) NMT Model: This model is trained to translate from target to source, helping to mitigate "exposure bias" where models are trained on human-generated text but tested on their own generated text.
  • Target-to-Source (T2S) NMT Model: This model directly assesses the adequacy of a translation by scoring the probability of the source sentence given the translated output.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Parallel Data:
    • UN Parallel Corpus (15.8M sentence pairs)
    • CWMT Corpora (9M sentence pairs)
    • News Commentary Corpus (332K sentence pairs)
  • Monolingual Data:
    • 8M sentences from "News Crawl: articles from 2017" (for Chinese-English)
    • 8M sentences from the XMU Corpus (for English-Chinese)
  • Validation and Test Sets:
    • news-dev2017 (validation)
    • newstest2017 (test)
    • newstest2018 (final evaluation)

The authors applied rigorous filtering criteria to the parallel data, including removing sentences with non-Chinese characters, duplicate pairs, and those with poor alignment scores, ensuring data quality.

Key Insights and Ideas

  • Transformer Dominance: The Transformer architecture consistently outperforms RNN-based models, even when accounting for parameter count differences. This reinforces its position as the de facto standard for high-performance NMT.
  • Asymmetric Impact of Augmentations: Backtranslation and reranking techniques do not benefit both translation directions equally. Backtranslation was more impactful for English-Chinese, while reranking showed greater gains for Chinese-English.
  • Translationese Effects: The authors hypothesize that the asymmetry in improvements might be due to "translationese" – stylistic artifacts introduced when text is translated. They suggest that the UN corpus might contain more original English documents, influencing how backtranslation and reranking perform.
  • Reranking Potential: While reranking with auxiliary models significantly improves scores, the gap between the reranker's performance and an "oracle" (perfect reranking) remains substantial, indicating considerable room for future research in developing more effective reranking features.

Key Results

The University of Maryland's systems achieved strong results on the WMT18 tasks:

  • Chinese-English: The best system reached a BLEU score of 24.4 on newstest2018. Improvements over the base Transformer model were +1.4 BLEU on newstest2017 through backtranslation, ensembling, and reranking.
  • English-Chinese: The best system achieved a BLEU score of 39.0 on newstest2018. This system outperformed the WMT17 best system by +2.1 BLEU on newstest2017. Augmentations led to a +2.2 BLEU improvement from backtranslation alone.

Limitations and Future Work

  • Reranking Gap: The authors acknowledge that the difference between their reranker's performance and the oracle score is significant, suggesting that current reranking features are not fully capturing translation quality. This points to a need for more sophisticated features and investigation into why better translations are ranked lower by the models.
  • Beam Size Optimization: While increasing beam size improved results for reranking, it also showed that the benefits plateau and can even degrade performance for the reranker itself, highlighting the complex interplay between search strategies and reranking.
  • Future Work: The paper implicitly suggests future work in developing more effective reranking features to close the gap with oracle scores and further exploring the asymmetric impact of various techniques across different language pairs.

Summary updated

BibTeX
@inproceedings{xu2018university,
  author = {Weijia Xu and Marine Carpuat},
  title = {{The University of Maryland's Chinese-English Neural Machine Translation Systems at WMT18}},
  booktitle = {WMT (shared Task)},
  year = {2018},
  url = {https://openreview.net/forum?id=jHoGFi34KFF},
}
Figure