University of Tartu’s Multilingual Multi-domain WMT19 News Translation Shared Task Submission

Andre Tättar, Elizaveta Korotkova, Mark FishelACL 2019

TL;DR

This paper presents the University of Tartu's submission to the WMT19 News Translation Shared Task, focusing on a single multilingual system designed to handle multiple language pairs and text domains. Their approach leverages a Transformer architecture with word factors to encode both target language and domain information, aiming for a more efficient and versatile translation system.

What Problem Does This Paper Address?

Traditional machine translation (MT) systems often specialize in a single language pair. While effective, this approach leads to a proliferation of models, making it resource-intensive to support many languages. The WMT News Translation Shared Task, in particular, involves numerous language pairs and diverse text domains (like parliamentary speeches, news articles, and press releases). This paper tackles the challenge of building a single, unified MT system capable of translating across multiple language pairs and adapting to different text domains within the WMT shared task constraints. The goal is to achieve comparable or better performance than specialized systems while offering greater flexibility and efficiency.

Methods

The core of the University of Tartu's submission is a Transformer-based neural MT system. They achieve multilinguality by incorporating an additional input that specifies the target language. Unlike some previous methods that embed this information directly into the input segment, they use "word factors" – a technique where the target language is represented as a factor of each input token.

Crucially, they extend this to handle multiple text domains by treating each domain as a separate "language." This creates a "double-multilingual" system where both the target language and the target domain are specified as word factors. This multi-domain approach is inspired by prior work suggesting that explicitly handling domain information can improve translation quality and efficiency compared to simply mixing corpora.

Beyond these core innovations, their system also incorporates common MT practices: data filtering, ensembling multiple models, and fine-tuning on back-translated monolingual data.

Datasets

The primary datasets used were the constrained parallel and monolingual data provided for the WMT19 News Translation Shared Task. The paper notes that due to time constraints, they focused on language pairs using the Latin script.

The data pipeline involved several steps:

  • Filtering: Removing empty/long sentences, those with too many non-alphanumeric characters, sentences with significant length differences, and those automatically identified as being in the wrong language.
  • Tokenization: Using the Moses Tokenizer.
  • Truecasing: Training separate truecasing models for each language.
  • SentencePiece: Training a SentencePiece model on a large combined text file to create a shared subword vocabulary of 90,000 units. This was used to handle different resource levels across language pairs (upscaling low-resource, downscaling high-resource).
  • Factoring: Generating source factors for target domain and target language.
  • Sharding: Dividing the large dataset into manageable shards of 1 million sentences for efficient processing.

For domain grouping, they defined categories like "News," "Subs," "Off," and "Other," consolidating smaller corpora. Monolingual data, primarily from News Crawl, was extracted for back-translation.

Key Insights and Ideas

  • "Double-Multilingual" System: Treating text domains as separate "languages" within a multilingual framework is a novel way to explicitly manage domain-specific translation nuances.
  • Word Factors for Control: Using word factors to encode both target language and domain information offers a flexible and granular way to control the translation output.
  • Fine-tuning Pitfalls: The fine-tuning step on back-translated data, intended to improve performance, unexpectedly led to a significant increase in wrong-language outputs, highlighting the challenges of domain adaptation and data quality in multilingual settings.

Key Results

The paper presents results for a baseline system trained on parallel data and a fine-tuned system trained on back-translated monolingual data.

  • The baseline system achieved respectable scores, but the fine-tuned system saw a dramatic increase in wrong-language outputs, jumping from 1.22% to 60.24% as measured by FastText.
  • Their ensemble model, combining baseline and fine-tuned snapshots, performed similarly to "Online-X" systems (like Google Translate) in human evaluations, but worse than single-language pair NMT systems.
  • The authors attribute the wrong-language output issue to potential errors in the monolingual data used for back-translation and the model itself producing incorrect language outputs during this process.

Limitations and Future Work

The authors acknowledge that their results were "modest," primarily due to the issues encountered during the fine-tuning stage. They explicitly state that the precise reasons for the failure of fine-tuning are still under investigation.

Future work directions include:

  • Incorporating more languages and domains.
  • Testing online continuous back-translation.
  • Experimenting with alternative methods for providing output language and domain information to the NMT model.
  • Investigating further explanations for the observed wrong-language output behavior.

Summary updated

BibTeX
@inproceedings{tattar2019university,
  author = {Andre Tättar and Elizaveta Korotkova and Mark Fishel},
  title = {{University of Tartu’s Multilingual Multi-domain WMT19 News Translation Shared Task Submission}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure