CUNI System for the WMT19 Robustness Task

Jindřich Helcl, Jindřich Libovický, Martin PopelACL 2019

TL;DR

This paper presents the CUNI system's approach to the WMT19 Robustness Task, focusing on improving machine translation (MT) quality for noisy, informal text. They demonstrate that a strong Transformer-based model, pre-trained on large general-domain data, benefits only slightly from fine-tuning on a noisy dataset, suggesting inherent robustness in well-trained models.

What Problem Does This Paper Address?

Traditional machine translation systems are typically evaluated on clean, formal text, often sourced from news articles. However, in real-world applications, MT systems frequently encounter informal text with spelling errors, grammatical mistakes, and other linguistic variations, commonly found in online content. Existing MT systems often struggle with this "noisy" input, leading to degraded translation quality. This paper addresses the challenge of building MT systems that are robust to such noise, specifically focusing on the WMT19 Robustness Task, which utilizes the Machine Translation of Noisy Text (MTNT) dataset.

Methods

The CUNI system employs a two-pronged approach:

  1. Strong Transformer Baseline: They leverage the "Transformer Big" model, a state-of-the-art architecture, pre-trained on a massive corpus of general-domain parallel data. This includes standard WMT15 News Translation data and additional synthetic data generated through back-translation. This pre-training aims to build an inherently robust model.
  2. Fine-Tuning on Noisy Data: After establishing the strong baseline, they fine-tune the model on the MTNT dataset. This fine-tuning process uses plain stochastic gradient descent with a carefully selected learning rate to adapt the model to the specific noise characteristics of the MTNT data without negatively impacting its performance on clean text.

Datasets

  • Training Data:
    • WMT15 News Translation parallel data.
  • Synthetic data from back-translation of French and English News Crawl corpora (years 2008-2014 for French, 2011-2017 for English).
  • Evaluation Data:
    • MTNT dataset (English-French and French-English) for training, validation, and testing.
  • News-test2014 (a standard WMT test set).
  • WMT19 Robustness Task blind test set.

The MTNT dataset is highlighted as a key benchmark, consisting of Reddit posts, which are inherently noisy and informal, differing significantly from standard news corpora.

Key Insights and Ideas

  • Inherent Robustness of Strong Models: A well-trained, large Transformer model already possesses a significant degree of robustness to noisy input, outperforming simpler RNN-based baselines by a considerable margin.
  • Diminishing Returns of Fine-Tuning: For already strong Transformer models, fine-tuning on a noisy domain dataset yields only marginal improvements. This suggests that the benefits of fine-tuning are more pronounced for weaker or less robust baseline systems.
  • Potential for Negative Transfer: Fine-tuning on the noisy MTNT dataset, especially for the French-to-English direction, can lead to a slight decrease in translation quality on general-domain (clean) data. This indicates a trade-off that needs careful management.
  • MTNT Dataset Peculiarities: The authors note that the MTNT training and validation splits have peculiar characteristics (e.g., validation source sentences starting with "Y"), which might make validation scores unreliable and could pose challenges for models trained on this split.

Key Results

  • The CUNI Transformer baseline significantly outperformed the RNN-based MTNT baseline on both WMT and MTNT test datasets.
  • Fine-tuning the RNN baseline resulted in substantial gains of 7-8 BLEU points.
  • Fine-tuning the Transformer baseline provided a smaller improvement of around 2 BLEU points in either direction.
  • The CUNI system achieved competitive results, outperforming some other submissions in the WMT19 Robustness Task, though it was surpassed by Naver Labs Europe, which used a similar architecture but incorporated corpus tags and synthetic noise generation.

Limitations and Future Work

  • Limited Fine-Tuning Benefit: The paper highlights that for strong Transformer models, the benefit of fine-tuning on noisy data is limited. This raises questions about the optimal strategies for adapting advanced models to specific noisy domains.
  • MTNT Dataset Issues: The identified peculiarities in the MTNT training/validation splits could affect model development and evaluation.
  • Comparison with Other Systems: While the paper compares its results to other WMT19 participants, details about the architectures and training strategies of some competing systems were not fully known at the time of submission, limiting direct comparative analysis.
  • Future Work (Implied): The authors' findings suggest that future research could focus on developing inherently more robust architectures or training methodologies that don't rely heavily on domain-specific fine-tuning, or on more sophisticated fine-tuning techniques that mitigate negative transfer. Investigating the impact of fine-tuning on the baseline model's performance on clean data would also be an interesting avenue.

Summary updated

BibTeX
@inproceedings{helcl2019cuni,
  author = {Jindřich Helcl and Jindřich Libovický and Martin Popel},
  title = {{CUNI System for the WMT19 Robustness Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure