The En-Ru Two-way Integrated Machine Translation System Based on Transformer

Doron YuACL 2019

TL;DR

This paper presents a two-way Russian-English machine translation system built upon the Transformer architecture, achieving competitive results in the WMT 2019 shared task. The authors detail their extensive data preprocessing, augmentation (including back-translation), and model ensembling strategies to improve translation quality.

What Problem Does This Paper Address?

Machine translation (MT) has seen significant advancements with neural network approaches, particularly the Transformer architecture. However, achieving high-quality translation, especially for linguistically complex language pairs like Russian and English, remains challenging. This paper addresses the problem of building a robust and accurate two-way (Russian-to-English and English-to-Russian) MT system for the WMT 2019 shared task, focusing on optimizing performance through careful data handling and model integration.

Methods

The core of the system is the Transformer architecture (Vaswani et al., 2017), a standard for modern neural MT. The authors augment this with several key techniques:

  • Data Preprocessing: This was a crucial step, involving:
  • Deduplication: Using SimHash to identify and remove near-duplicate sentence pairs.
  • Length Filtering: Ensuring sentence length ratios between source and target languages were within a reasonable range (1/2 to 2/1).
  • Sentence Alignment: Employing the giza++ tool and an IBM model 1 with EM algorithm to generate word alignments and filter poorly aligned pairs.
  • Data Augmentation (Back-Translation): To increase the training data, they used back-translation. A preliminary translation model was trained, then used to translate monolingual target data into synthetic source data, creating new parallel pairs. This synthetic data was then filtered and combined with the original data.
  • Model Training Strategies: Several models were trained and evaluated:
  • Baseline Model: Trained on raw, unfiltered data.
  • Filtered Model: Trained on data after extensive preprocessing.
  • Back Translation Model: Trained on data augmented with back-translated pairs.
  • Fine-tuning Model: A trained model was further fine-tuned on a small, high-similarity corpus.
  • Ensemble Model: Combining predictions from multiple trained models.
  • Average Model: Averaging parameters from multiple models.
  • Nbest and Rerank Model: Selecting top-N hypotheses and reranking them using beam search and language model scores.

Datasets

The primary dataset used was the official WMT 2019 training data, which initially comprised approximately 38 million parallel Russian-English sentence pairs. After filtering, this was reduced to about 33 million. Additionally, 10 million monolingual Russian and English sentences were used for back-translation. The paper does not introduce new benchmarks but rather participates in the existing WMT shared task.

Key Insights and Ideas

  • Data quality is paramount: The paper strongly emphasizes that rigorous data preprocessing and filtering significantly improve translation quality, even with a standard Transformer model.
  • Back-translation is effective for data augmentation: This technique proved valuable in expanding the training corpus and enhancing model robustness, especially when external parallel data is restricted.
  • Ensembling and reranking are crucial for final performance: Combining multiple models and employing sophisticated reranking strategies yielded the best results, demonstrating that no single model is optimal.
  • Linguistic characteristics matter: The authors acknowledge the inherent difficulty of Russian-English translation due to linguistic differences and tailored their preprocessing accordingly.

Key Results

The paper reports the following BLEU scores for their final system:

  • Russian to English (Ru-En): 38.7 (ranking 5th in the WMT 2019 shared task)
  • English to Russian (En-Ru): 27.8 (ranking 10th in the WMT 2019 shared task)

These results represent a significant improvement over baseline models, with the reranked model showing gains of 2-3 BLEU points. The authors note that their achieved scores are competitive and achieved within limited time and hardware resources.

Limitations and Future Work

  • Time and Hardware Constraints: The authors acknowledge that not all experiments were fully detailed or refined due to limitations in time and hardware resources.
  • Potential Duplication: Back-translation can introduce duplication between synthetic and original corpora, which they attempted to mitigate but acknowledge as a potential issue.
  • Future Work: The authors express a desire to continue improving their system, aiming for even better results and exploring further optimization strategies to achieve higher rankings in future evaluations. They also mention the potential for further refinement of ensemble and reranking strategies.

Summary updated

BibTeX
@inproceedings{yu2019en,
  author = {Doron Yu},
  title = {{The En-Ru Two-way Integrated Machine Translation System Based on Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure