TL;DR
The eTranslation team at the European Commission participated in the WMT 2019 News Translation Task by developing machine translation systems for four language pairs. Their approach focused on efficiency and transferability to production systems, demonstrating that decent performance can be achieved without overly complex architectures, even under resource constraints.
What Problem Does This Paper Address?
The European Commission's eTranslation project aims to facilitate cross-language communication within the EU by providing machine translation services. While historically focused on official EU documents, there's a growing need to handle a wider variety of text domains, including news. The WMT 2019 News Translation Task presented an opportunity for the eTranslation team to explore best practices for building robust news translation systems within the constraints of their production environment, which is characterized by limited computational resources. This paper details their efforts to adapt existing techniques and identify effective strategies for this specific challenge.
Methods
The eTranslation team employed a pragmatic approach, leveraging the Marian NMT toolkit and focusing on standard, efficient techniques. Key methods included:
- Data Cleaning and Filtering: Rigorous preprocessing of parallel and monolingual data was crucial. This involved language identification, segment deduplication, filtering based on token ratio and length, and exclusion of segments without alphabetic characters. They also applied dual conditional cross-entropy filtering to reduce noise in large parallel corpora like ParaCrawl and CommonCrawl.
- Baseline Transformer Models: They trained base transformer models for all language pairs, opting for dynamic batching and tied embeddings. Training was stopped early if validation performance plateaued to save resources.
- Synthetic Data Generation (Back-translation): Back-translation was a primary method for data augmentation. They experimented with various amounts of synthetic data, finding that carefully selected and filtered back-translated data significantly improved performance.
- Fine-tuning with In-domain Data: For language pairs with available historical test sets, they fine-tuned models on domain-specific data to adapt them to the news domain.
- Ensembling: Simple ensembles of models trained with different random seeds were used to boost performance. They also experimented with ensembling translation models with language models.
- Topic Modeling: For French-German, they used guided topic modeling to select relevant data for back-translation, which proved particularly effective.
Datasets
The team utilized a combination of publicly available datasets and those specific to the WMT 2019 task:
- Parallel Corpora: Europarl v9, Common Crawl, News Commentary v14, Rapid Corpus, Wiki Titles v1, Yandex.
- Monolingual Corpora: News Crawl (various years), ParaCrawl, CommonCrawl.
- Development and Test Sets: Provided by the WMT 2019 News Translation Task, along with historical test sets from previous WMT tasks (e.g., 2008-2014 for Fr-De).
They noted that the quality of ParaCrawl v3 was notably higher than previous versions, requiring less aggressive filtering.
Key Insights and Ideas
- Efficiency Over Complexity: The core idea was to achieve good performance without resorting to prohibitively deep or complex architectures, making the systems more practical for production.
- Data Quality is Paramount: Rigorous data cleaning and filtering, especially for large, noisy datasets like ParaCrawl, was essential. The effectiveness of filtering ParaCrawl v3 was highlighted.
- Back-translation is Powerful, but Nuanced: While back-translation generally improved performance, the amount and source of synthetic data mattered. Topic-selected back-translation for Fr-De yielded particularly strong results.
- Fine-tuning for Domain Adaptation: Fine-tuning on in-domain data (news test sets) proved beneficial, though it could sometimes harm performance on the target test set if the fine-tuning data was too specific or misaligned.
- Simple Ensembles Work: Basic ensembling of models trained with different seeds provided a reliable boost in performance.
Key Results
The eTranslation team submitted one model for each of the four language pairs (English-German, French-German, English-Lithuanian, and Russian-English). Their systems generally performed well, placing in the first half of the rankings for most language pairs.
- English-German: Model 6 (fine-tuned) achieved a BLEU score of 42.4 on the 2019 test set, a significant improvement over the baseline (38.1).
- French-German: Model 4 (incorporating topic-selected back-translated data) achieved a BLEU score of 33.5 on the 2019 test set, outperforming the baseline (26.1).
- English-Lithuanian: Model 6 (an ensemble) achieved a BLEU score of 17.1 on the 2019 test set, a substantial gain from the baseline (11.4).
- Russian-English: Model 3 (fine-tuned on domain-specific data) achieved a BLEU score of 37.4 on the 2019 test set, an improvement over the baseline (32.4).
Limitations and Future Work
- Resource Constraints: The primary limitation was the low-resource environment, which restricted the scope of experimentation, particularly with hyperparameter tuning and large-scale model architectures.
- Unconstrained Models: Experiments with unconstrained models using the large Euramis dataset did not yield improvements and sometimes even degraded performance, suggesting that domain mismatch was a significant issue.
- Fine-tuning Risks: Fine-tuning on domain-specific data, while generally beneficial, could lead to performance drops on the target test set if not carefully managed.
- Future Work: The paper implicitly suggests that further exploration of data selection strategies, more sophisticated ensembling techniques, and potentially exploring larger models (if resources permit) could lead to further improvements. The focus on transferability to production systems also implies a continuous effort to integrate effective techniques into their operational services.
Summary updated