OPPO’s Machine Translation System for the IWSLT 2020 Open Domain Translation Task

Qian Zhang, Xiaopu Li, Dawei Dang, Tingxun Shi, Di Ai, Zhengshan Xue, Jie HaoACL 2020

TL;DR

This paper presents OPPO's machine translation system for the IWSLT 2020 Chinese-Japanese open domain task. They leverage a Transformer architecture, enhanced by extensive data preprocessing, rule-based filtering, and iterative back-translation and knowledge distillation techniques to achieve strong results.

What Problem Does This Paper Address?

The paper tackles the challenge of building a high-quality machine translation system for the Chinese-Japanese language pair, specifically within an open-domain setting. This task is complicated by the inherent differences in the languages (e.g., logographic vs. agglutinative structures) and the common issue of limited high-quality parallel data. While Transformer models are powerful, they require substantial data, making it crucial to effectively utilize available parallel corpora and explore methods for augmenting them, especially when dealing with noisy, large-scale datasets. The authors also investigate the potential of character-based translation models due to the shared linguistic heritage of Chinese and Japanese.

Methods

The core of the system is the Transformer architecture. To prepare the data, they employ a multi-stage data preprocessing pipeline that includes:

  • Rule-based filtering: This involves steps like converting text to lowercase, handling Chinese character simplification/kanji conversion, normalizing punctuation, unescaping HTML, removing HTML tags, and cleaning up extra spaces.
  • Segmentation: They use PKUSEG for Chinese and MeCab for Japanese to segment sentences into words, as these languages don't use spaces for word boundaries.
  • Sentence pair filtering: This is a crucial step that removes pairs based on criteria like excessive non-sense symbols, abnormal length ratios, duplicates, exact replicas, similar beginnings/endings, and insufficient Chinese/Japanese word proportions.
  • Alignment Information Based Filtering: They use fast_align to further filter sentence and word-level alignments, significantly reducing noisy data.

For data augmentation, they extensively use back-translation (BT) and knowledge distillation (KD). This involves training models on existing data, using them to translate monolingual data, and then retraining on the combined datasets. They iterate this process multiple times.

Additionally, they explore character-based models by splitting subwords into characters in different ways (CJK characters only, CJK + katakana, CJK + all kana) to see if this approach offers advantages over subword-based models, given the shared character set.

Finally, they employ ensemble modeling and reranking of the final outputs to further improve performance.

Datasets

The primary dataset used is from the IWSLT 2020 Open Domain Translation Task. This includes:

  • Original corpus: A very large, crawled dataset (15.7 million documents, ~941M Japanese sentences, ~928M Chinese sentences) that is not parallel.
  • Pre-filtered file: Contains 161.5 million "parallel" sentences, but still requires significant cleaning.
  • Filtered file: An officially aligned subset of the pre-filtered file, containing 19 million sentences.
  • Existing parallel file: A provided dataset of 1.96 million sentence pairs.

They combine the filtered and existing parallel files to create a 20 million pairs dataset for initial training. After extensive filtering and augmentation, they work with approximately 5.4 million sentence pairs for the baseline and subsequent experiments. They also utilize monolingual corpora for back-translation.

Key Insights and Ideas

  • Data quality is paramount: The paper strongly emphasizes that even with powerful models like Transformers, the quality and cleanliness of the training data are critical. Extensive, multi-stage filtering and preprocessing are essential for achieving good results, especially with noisy, large-scale datasets.
  • Iterative augmentation pays off: Multiple rounds of back-translation and knowledge distillation, combined with ensemble methods, significantly boost performance. This iterative refinement process is key to leveraging monolingual data effectively.
  • Character-based translation shows promise but needs more research: While the shared character system between Chinese and Japanese suggests potential for character-based models, their experiments did not yield improvements over subword models. The authors hypothesize that the large amount of subword-based synthetic data might have skewed the distribution. This area warrants further investigation.
  • Careful post-processing matters: Even after training, a final post-processing step to handle full-width vs. half-width characters for numbers and Latin words led to a notable improvement, highlighting the importance of fine-tuning output.

Key Results

  • The final system achieved a BLEU score of 32.9 for jazh and 30.1 for zhja on the official dev set.
  • After filtering by alignment information and using the baseline model, they achieved a BLEU score of 36.3 for zh-ja and 33.2 for ja-zh, an improvement of +1.7 and +0.6 respectively over the initial baseline.
  • Through iterative back-translation and ensemble methods, their final system reached 40.1 BLEU for zh-ja and 36.6 for ja-zh on the dev set, representing significant gains over the baseline.
  • A contrastive experiment showed that converting numbers and Latin words to full-width characters improved the BLEU score by nearly 5 points.

Limitations and Future Work

  • Character-based models: The character-based translation experiments did not yield improvements and even showed degradation in some cases. The authors suggest that the synthetic data generated by subword models might have negatively impacted the latent data distribution, and further work is needed to optimize this approach.
  • Mining unaligned data: While they explored mining unaligned data and achieved a minor BLEU improvement (+0.6), this data was not fully integrated into the main task experiments due to time constraints. Further work could explore better integration strategies.
  • Open domain challenges: The paper acknowledges that open-domain translation is inherently challenging, and while their system performed well, there's always room for improvement in handling diverse topics and linguistic nuances.
  • Future work: The authors plan to continue optimizing data mining and filtering techniques, explore character-based models further, and investigate other methods to improve the translation procedure and models.

Summary updated

BibTeX
@inproceedings{zhang2020oppos,
  author = {Qian Zhang and Xiaopu Li and Dawei Dang and Tingxun Shi and Di Ai and Zhengshan Xue and Jie Hao},
  title = {{OPPO’s Machine Translation System for the IWSLT 2020 Open Domain Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure