System Description: The Submission of FOKUS to the WMT 19 Robustness Task

Cristian GrozeaACL 2019

TL;DR

This paper presents Fraunhofer FOKUS's submission to the WMT 2019 robustness task, exploring how well standard machine translation models handle noisy text and if intentionally introducing typos during training can improve robustness. They found that while standard models show some resilience, specific robustness training, especially with injected typos, is crucial for significant improvements.

What Problem Does This Paper Address?

The WMT robustness task aims to evaluate how well machine translation (MT) systems perform when faced with noisy or imperfect input text, a common scenario in real-world applications. Traditional MT systems are often trained on clean, curated datasets, which can lead to a significant drop in performance when encountering variations like typos, grammatical errors, or informal language. This paper investigates two key questions: 1) How robust are well-performing MT models that are trained on clean data? and 2) Can intentionally introducing "typos" into the training data make models more robust to such noise?

Methods

The FOKUS team developed two main approaches for their submissions:

  • Baseline Model (EN-FR, FR-EN): For the English-French and French-English language pairs, they used a standard Sequence-to-Sequence model with shallow networks, trained on the UFAL medical corpus. This corpus was chosen for its relevance to a related biomedical translation task. They employed BPE (Byte Pair Encoding) for subword tokenization, using FastBPE for efficiency. Training was configured to stop based on validation set performance to prevent overfitting.
  • Robustness-Focused Model (JA-EN): For the Japanese-English submission, they used the same model architecture and training procedure as the baseline. However, the key difference was the preprocessing step, which involved intentionally introducing typos into the training data. These typos were designed to mimic two common types of errors: missing letters and duplicated letters.

Datasets

  • Training Data:
    • UFAL Medical Corpus: Used for the EN-FR and FR-EN baseline models. The "Subtitles" portion of this corpus was removed due to perceived lower quality and less medical relevance.
  • UFAL Medical Corpus (with injected typos): Used for the JA-EN submission, where typos were intentionally added to the training data.
  • Validation Data:
    • Khresmoi: Used for validation, noted as not being included in the UFAL corpus.
  • Evaluation Data: The paper refers to the WMT 2019 robustness task evaluation data for all language pairs.

Key Insights and Ideas

  • Standard models have some inherent robustness: Even models trained on clean data show a degree of resilience to noisy text, though performance can still degrade.
  • Intentional typo injection is effective: Introducing specific types of noise (missing/duplicated letters) into the training data for the JA-EN pair demonstrably improved its robustness.
  • Domain mismatch can impact robustness: The choice of training data (biomedical corpus) might not perfectly align with the evaluation data (Reddit text), potentially affecting performance, especially for tasks involving informal language and first-person narratives.
  • "Shallow" models can be surprisingly effective: The use of relatively simple Sequence-to-Sequence models with shallow networks, trained on relevant data, yielded competitive results.

Key Results

  • EN-FR and FR-EN Baselines: These models surpassed previous baseline results from the MTNT paper by 2.5 (EN-FR) and 4 (FR-EN) BLEU points on the biomedical task. However, their performance on the robustness task showed a significant drop, particularly for EN-FR (8.3 BLEU points decrease).
  • JA-EN Robustness: The JA-EN submission, which incorporated typo injection during training, had a very low overall BLEU score (7.3 cased, 6.4 uncased). The authors acknowledge this was likely due to insufficient training data quantity and an error where typos were introduced into the target text as well as the source.
  • Qualitative Observations: The FR-EN and EN-FR models, while scoring better, sometimes translated too literally, missed nuances, or failed to process coreference correctly, indicating that even with some robustness, deeper understanding issues persist.

Limitations and Future Work

  • Data Quantity for JA-EN: The authors explicitly state that the training data for JA-EN was likely insufficient for training a reliable translation model.
  • Error in JA-EN Training: The unintentional introduction of typos into the target text during JA-EN training is a significant limitation that likely skewed results.
  • Domain Mismatch: The use of a biomedical corpus for training might not be optimal for evaluating robustness on more general or informal text like Reddit.
  • Future Work: The paper implicitly suggests that further research into more sophisticated methods for injecting diverse types of noise and exploring larger, more varied training datasets would be beneficial for building truly robust MT systems. Investigating how to better handle nuances and context in noisy text remains an open challenge.

Summary updated

BibTeX
@inproceedings{grozea2019system,
  author = {Cristian Grozea},
  title = {{System Description: The Submission of FOKUS to the WMT 19 Robustness Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure