NICT’s Supervised Neural Machine Translation Systems for the WMT19 Translation Robustness Task

Raj Dabre, Eiichiro SumitaACL 2019

TL;DR

This paper tackles the challenge of making Neural Machine Translation (NMT) systems robust to noisy text, a common issue in real-world applications like social media. The authors adapted existing Transformer NMT models using domain adaptation techniques, specifically Mixed Fine-Tuning (MFT), and explored bi-directional models to improve performance on noisy Japanese-English translation.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized translation, but its performance significantly degrades when faced with noisy or informal text, such as that found on social media. This noise can manifest as spelling mistakes, code-switching, or random characters, and it's often paired with a scarcity of relevant training data. The WMT19 Translation Robustness Task specifically aims to evaluate and improve NMT systems' ability to handle such imperfect input. This paper addresses the gap in NMT robustness by exploring domain adaptation techniques to make models more resilient to these real-world challenges.

Methods

The core of the authors' approach is domain adaptation applied to the Transformer NMT model. They employed two main strategies:

  • Bi-directional NMT: Instead of the standard uni-directional Transformer, they trained bi-directional models. This architecture is known to be more robust, especially in low-resource settings. They achieved this by concatenating Japanese-English and English-Japanese training corpora and prepending special tokens ("2en" or "2ja") to indicate the translation direction.
  • Mixed Fine-Tuning (MFT): This is a domain adaptation technique designed to mitigate the risk of overfitting when fine-tuning a model on a small, in-domain dataset. MFT involves fine-tuning an out-of-domain model on a combination of both the original out-of-domain data and an oversampled version of the in-domain data. The authors highlight that MFT avoids the risk of quick overfitting associated with standard fine-tuning.

Datasets

The paper utilized the following datasets:

  • Out-of-domain (non-noisy) datasets: KFTT, JESC, and TED Talks. The total number of sentence pairs for these datasets was 3,900,772.
  • In-domain (noisy) datasets:
    • Japanese-English: 6,506 training sentence pairs.
  • English-Japanese: Initially 5,775 pairs, which were increased to 10,060 pairs after a manual paragraph splitting method was applied to handle long sentences.

The MTNT (Michel and Neubig, 2018) test suite is mentioned as a benchmark for testing MT models robust to noisy text. The authors also refer to a non-blind test set provided with the training data for evaluation.

Key Insights and Ideas

  • Noise is a significant challenge for NMT: Even with advanced models like the Transformer, noisy text drastically impacts translation quality, especially when in-domain training data is limited.
  • Bi-directional models offer robustness: Training NMT models in a bi-directional manner can inherently improve their resilience to imperfect input, particularly in low-resource scenarios.
  • Mixed Fine-Tuning is effective for domain adaptation: MFT provides a way to adapt models to specific domains without the severe overfitting risks associated with standard fine-tuning on small datasets. It's presented as a key driver for performance improvements.
  • Standard Transformer limitations with noise: The authors observed that the Transformer model they used struggled with certain types of noise, such as emojis, which were often omitted in translations.
  • Morphological segmentation matters: Issues with morphological segmentation (using tools like KyTea) were identified as a potential reason for poor performance, suggesting that better pre-processing is crucial.

Key Results

  • English-Japanese Translation: The submitted system achieved a cased BLEU score of 11.1.
  • Japanese-English Translation: The submitted system achieved a BLEU score of 8.1.
  • Comparison of Approaches (Table 2):
    • Bidirectional FT (Fine-Tuning): Achieved BLEU scores of 9.6 (Ja->En) and 10.5 (En->Ja) on the non-blind test set.
  • Bidirectional MFT (Mixed Fine-Tuning): Achieved BLEU scores of 9.2 (Ja->En) and 13.4 (En->Ja) on the non-blind test set.
  • MFT vs. Fine-Tuning: MFT was found to be either comparable to or significantly better than regular fine-tuning, particularly improving English-to-Japanese translation by approximately 2 BLEU points. This improvement was the primary reason for choosing MFT for the final submission.

Limitations and Future Work

  • Limitations:
    • The Transformer model used was not well-suited to handle all types of noisy text, specifically mentioning the omission of emojis.
  • The default KyTea model's morphological segmentation was not optimal, contributing to translation quality issues.
  • The improvements obtained were described as "modest" due to the noisy nature of the in-domain training data.
  • Future Work:
    • Incorporating better pre-processing mechanisms into the NMT implementation.
  • Exploring more advanced domain adaptation techniques.
  • Investigating data augmentation techniques to create more robust translation systems.
  • The authors attempted back-translation on the MTNT dataset but were unable to achieve satisfactory results, suggesting this is an area for further exploration.

Summary updated

BibTeX
@inproceedings{dabre2019nicts,
  author = {Raj Dabre and Eiichiro Sumita},
  title = {{NICT’s Supervised Neural Machine Translation Systems for the WMT19 Translation Robustness Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure