Naver Labs Europe’s Systems for the WMT19 Machine Translation Robustness Task

Alexandre Berard, Ioan Calapodescu, Claude RouxACL 2019

TL;DR

This paper presents Naver Labs Europe's winning systems for the WMT19 Machine Translation Robustness Task. They tackled the challenge of noisy, informal text common on social media by developing robust NMT models that significantly outperformed others, achieving first place in all translation directions. Their success stemmed from a combination of meticulous data filtering, effective domain adaptation techniques, and clever handling of noisy text characteristics.

What Problem Does This Paper Address?

Modern Neural Machine Translation (NMT) systems have achieved remarkable success, often approaching human-level performance on clean, well-structured text. However, these systems are notoriously "brittle," meaning they degrade significantly when faced with the kind of noise commonly found in real-world data, especially on social media. This noise includes informal language, spelling and grammatical errors, inconsistent capitalization, slang, emojis, and even code-switching. The WMT19 Machine Translation Robustness Task specifically aimed to address this by focusing on translating noisy text from Reddit, a popular social media platform. The challenge was to build NMT models that could not only translate accurately but also maintain robustness to these common linguistic imperfections.

Methods

The authors employed a multi-pronged approach to build their robust NMT systems:

  • Data Filtering and Pre-processing: A significant effort was dedicated to cleaning the training data. This involved:
  • Language Identification (LID): Removing sentence pairs where the source or target language was misidentified.
  • Length Filtering: Discarding pairs with extreme length ratios, which often indicate errors.
  • Common Crawl Filtering: Identifying and removing specific types of "hallucinations" (generated text unrelated to the source) that were traced back to the Common Crawl dataset. They developed an "attention-based filtering" method to detect and remove these problematic sentence pairs.
  • SentencePiece Tokenization: Using SentencePiece for subword segmentation, which is language-independent and handles various characters, including emoticons, more robustly. They also explored pre-tokenization for Japanese to improve segmentation.
  • Robustness Techniques:
    • Inline Casing: To handle inconsistent capitalization, they introduced special tokens (<U>, <T>) to mark uppercase and title-cased words within the input, treating case as an additional feature.
  • Placeholder Replacement: Emojis and Reddit user/subreddit names, which NMT models struggle with, were replaced with special placeholder tokens (<emoji>, <user>, <reddit>) during training and then mapped back to their original forms or appropriate translations at test time.
  • Natural Noise Injection: They created noisy versions of words by applying edit distance rules (swaps, repetitions) and then augmented the training data with these noised source sentences.
  • Domain Adaptation:
    • Back-Translation: Leveraging large amounts of monolingual data from sources like "news-discuss" and "news-crawl" to generate synthetic parallel data, which is crucial for adapting models to new domains or low-resource scenarios.
  • Corpus Tags: Prepending tags to source sentences to indicate their origin (e.g., <MTNT>, <BT> for back-translated data). This allowed the model to learn domain-specific translation styles and improved performance on out-of-domain data.

Datasets

The primary dataset used for this task was MTNT (Machine Translation of Noisy Text), a real-world noisy corpus derived from Reddit. The authors also utilized:

  • WMT15 News Translation Task data: For French-English translation.
  • KFTT, TED, and JESC datasets: For Japanese-English translation.
  • Monolingual data: From "news-discuss" and "news-crawl" for back-translation.

The MTNT dataset was specifically curated to include informal language, spelling mistakes, and other orthographic variations characteristic of social media.

Key Insights and Ideas

  • Data quality is paramount: The paper strongly emphasizes that meticulous data filtering and cleaning, especially for noisy sources like Common Crawl, can yield significant improvements in translation quality and robustness, often more so than complex model architectures alone.
  • Attention patterns reveal errors: The observation that hallucinated outputs often correlate with a peculiar attention matrix (concentrated on the source EOS token) was a key insight for developing their attention-based filtering method.
  • Handling social media specifics: Directly addressing elements like emojis, slang, and inconsistent casing with dedicated techniques (placeholders, inline casing) proved effective, even if their impact on BLEU was sometimes subtle due to their rarity in test sets.
  • Back-translation and corpus tags are powerful for domain adaptation: Combining back-translation with explicit corpus tags allowed the models to effectively leverage monolingual data and adapt to the specific domain of noisy social media text.

Key Results

The systems developed by Naver Labs Europe achieved first place in the WMT19 Machine Translation Robustness Task across all translation directions (French-English and Japanese-English, in both directions). Their ensemble models significantly outperformed other participants, with BLEU score improvements ranging from +0.7 to +3.1 compared to the next best results. While specific robustness techniques like inline casing and placeholder replacement had a minor impact on BLEU scores (likely due to the low frequency of these specific noise types in the test set), the overall system, built on careful data filtering and domain adaptation, demonstrated superior performance.

Limitations and Future Work

The authors acknowledge that the impact of some specific robustness techniques (like inline casing and emoji handling) was difficult to measure with BLEU on the provided test sets, as these noise phenomena were relatively rare. They suggest that measuring these improvements would be more effective on test sets where these phenomena have been artificially increased, such as an all-uppercase test set or the natural noise datasets used in prior work. Future work could explore more sophisticated methods for handling the diverse and evolving nature of social media language.

Summary updated

BibTeX
@inproceedings{berard2019naver,
  author = {Alexandre Berard and Ioan Calapodescu and Claude Roux},
  title = {{Naver Labs Europe’s Systems for the WMT19 Machine Translation Robustness Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure