TL;DR
The MLLP-UPV team tackled the WMT18 German-English news translation task by building a Transformer-based neural machine translation (NMT) system. They significantly improved performance by carefully filtering the large, noisy ParaCrawl corpus and augmenting the training data with synthetically generated German sentences.
What Problem Does This Paper Address?
The paper addresses the challenge of building high-quality machine translation systems, specifically for the German-English news translation task in WMT18. A major hurdle in this task is the availability and quality of parallel training data. While neural machine translation (NMT) has advanced significantly, its performance is heavily reliant on large, clean parallel corpora. The WMT18 shared task introduced the massive ParaCrawl corpus, which, while offering a huge quantity of data, is known to be significantly noisier than previously available datasets. This noise can degrade translation quality if not handled properly. The authors also aimed to work within "constrained" conditions, meaning they had to be mindful of how they used external data.
Methods
The MLLP-UPV team employed several key techniques:
- Transformer Architecture: They adopted the Transformer architecture, a standard for state-of-the-art NMT, for its efficiency and performance. They used the "base" configuration for faster training.
- Corpus Preprocessing: Standard preprocessing steps were applied, including punctuation normalization, tokenization, cleaning, and truecasing using Moses scripts. They also removed sentences with characters outside a specific Latin UTF range to reduce vocabulary size and focus on relevant languages.
- Corpus Filtering: This was a crucial step. They filtered the noisy ParaCrawl corpus using a novel approach: training two separate 9-gram character-based language models (one for German, one for English) on a clean development set. They then computed the perplexity of each sentence pair in the full WMT18 dataset using these models. Sentence pairs with lower perplexity (indicating they were more "in-domain" and less noisy) were prioritized. The geometric mean of the German and English perplexity scores was used to rank sentence pairs.
- Data Augmentation with Synthetic Sentences: To further improve performance and adhere to "constrained" conditions, they generated synthetic German source sentences from the provided English monolingual corpora. This was achieved by training an English-to-German NMT system and using it to translate a large sample of English sentences.
- Ensembling: The final submission was an ensemble of four independently trained Transformer models, combining their individual probabilities for improved robustness and performance.
Datasets
- Training Data:
- WMT18 parallel dataset (including News Commentary v13, Rapid, Common Crawl, Europarl v7, and ParaCrawl).
- ParaCrawl corpus (approx. 36.4 million sentence pairs).
- Monolingual corpora (English and German) for synthetic data generation.
- Evaluation Data:
- newstest2015 (development set)
- newstest2017 (test set)
- newstest2018 (test set)
Key Insights and Ideas
- ParaCrawl is Noisy, but Usable with Filtering: The paper strongly demonstrates that simply using the massive ParaCrawl corpus "as is" leads to a degradation in translation quality. However, their character-based language model perplexity filtering technique effectively extracts valuable data from ParaCrawl, leading to significant improvements.
- Perplexity as a Proxy for Noise: Using language model perplexity, especially character-based, is an effective and relatively simple method to identify and filter out noisy sentence pairs from large, uncurated parallel corpora.
- Synthetic Data Augmentation is Effective: Generating synthetic source sentences from monolingual target data is a viable strategy to increase the amount of in-domain training data and boost translation performance, especially under constrained conditions.
- Transformer Efficiency: The Transformer "base" model offers a good balance between translation quality and training time, making it suitable for participation in time-sensitive shared tasks.
Key Results
- Filtering Improvement: Using the filtered 10M sentence pair subset of WMT18 (excluding ParaCrawl) improved BLEU scores by 2.5 and TER by 1.7 over the baseline (which used only the cleaner 6M pairs).
- Synthetic Data Impact: Adding 20M synthetic sentence pairs, along with oversampling the filtered bilingual data, further boosted performance, resulting in a single system achieving 35.9 BLEU and 51.2 TER on the test set.
- Ensemble Performance: The final ensemble of four models achieved 36.2 BLEU and 51.0 TER on newstest2017, and 45.1 BLEU and 40.8 TER on newstest2018.
- Training Time: Training a single system with the final configuration took approximately 120 hours on a single Nvidia GeForce GTX 1080 Ti GPU.
Limitations and Future Work
- Limitations of Perplexity Filtering: The authors acknowledge that their perplexity-based filtering method cannot detect sentence pairs where both sentences are valid but not actual translations of each other.
- Future Work:
- Combining the perplexity score with a quick translation model score to better detect misalignments.
- Exploring other data selection approaches, such as cross-entropy difference or dynamic data selection methods.
- Investigating different ratios of synthetic versus original data for augmentation.
- Exploring multi-GPU training for potentially better translation quality.
Summary updated