Bering Lab’s Submissions on WAT 2021 Shared Task

Heesoo Park, Dongjun LeeACL 2021

TL;DR

Bering Lab tackled the 8th Workshop on Asian Translation (WAT 2021) by focusing on domain adaptation for machine translation of technical documents. Their approach, centered on building specialized NMT systems using curated in-domain corpora, led to top rankings in 8 out of 14 tasks, demonstrating significant BLEU score improvements.

What Problem Does This Paper Address?

Machine translation (MT) systems often struggle with highly specialized domains, such as patent documents or software documentation, due to the prevalence of technical jargon and unique sentence structures not commonly found in general-purpose text. The WAT 2021 shared task aimed to advance Asian language translation, specifically focusing on these challenging domains. Bering Lab's work addresses the critical need for robust MT systems that can accurately translate technical content, bridging the gap between general MT capabilities and domain-specific requirements.

Methods

The core of Bering Lab's approach was domain adaptation through meticulous data curation and training. For both the JPO Patent Corpus (JPC2) and the NICT-SAP IT domain tasks, they focused on building Neural Machine Translation (NMT) systems using specialized corpora.

Key techniques included:

  • Web Crawling and Data Augmentation: For JPC2, they crawled patent documents from the web to create additional parallel corpora for English-Japanese, Chinese-Japanese, and Korean-Japanese.
  • Sentence Alignment and Similarity Scoring: To build high-quality parallel corpora, they aligned sentences and used sentence-level similarity scores to filter noisy data.
  • Leveraging Existing Datasets: For NICT-SAP, they utilized the OPUS dataset, specifically weighting IT domain corpora during training.
  • Transformer Architecture: They employed the Transformer model (Vaswani et al., 2017), a widely recognized and powerful architecture for NMT.
  • Data Preprocessing and Tokenization: They used SentencePiece for subword tokenization, with vocabulary sizes tailored to each language pair and domain.
  • Model Ensembling: For inference, they used ensembles of multiple trained models to further boost performance.

Datasets

The paper utilized two main datasets for training and evaluation:

  • JPO Patent Corpus (JPC2): This corpus focuses on patent descriptions for Japanese ↔ English, Chinese ↔ Japanese, and Korean ↔ Japanese translation. Bering Lab augmented the provided training data with their own crawled patent documents.
  • NICT-SAP IT Domain Corpus: This dataset comprises software documentation for English ↔ Hindi, English ↔ Indonesian, English ↔ Malaysian, and English ↔ Thai translation. They trained on the OPUS dataset, with a focus on IT-specific corpora.

Key Insights and Ideas

  • Domain-Specific Data is King: The most significant insight is the profound impact of using highly relevant, in-domain data. Even with a strong base model like the Transformer, performance on technical translation tasks dramatically improves when trained on specialized corpora.
  • Careful Data Curation Pays Off: Simply having more data isn't enough. The paper highlights the importance of cleaning noisy data and aligning sentences effectively to create high-quality parallel corpora, which directly translates to better translation quality.
  • Weighted Training for Mixed Corpora: When combining general and domain-specific data (like with OPUS for NICT-SAP), intelligently weighting the domain-specific data is crucial for achieving optimal results.
  • Ablation Study Reveals Nuances: The ablation study on the Ja → Ko task showed that while additional data generally helps, performance can degrade on manually created test sets, suggesting that the quality and nature of the "additional" data matter.

Key Results

Bering Lab achieved remarkable success in the WAT 2021 shared tasks:

  • Top Rankings: They secured the 1st place in 8 out of 14 tasks.
  • Significant BLEU Score Improvements:
    • For JPC2, they achieved up to a 2.87 absolute improvement in BLEU score for Japanese to Korean translation.
  • For NICT-SAP, they saw an improvement of 8.79 for Indonesian to English translation.
  • State-of-the-Art Performance: Their models achieved new state-of-the-art results on four out of six JPC2 tasks and ranked first on four out of eight NICT-SAP IT domain tasks.
  • Specific Task Performance:
    • In JPC2, they ranked first in all tasks where Japanese was the input language.
  • For NICT-SAP, their models ranked first on sub-tasks where English was the input language, with notable improvements for English ↔ Indonesian.

Limitations and Future Work

  • Data Scarcity for Some Pairs: For the English ↔ Thai sub-task in NICT-SAP, the authors note that their model performed relatively poorly due to the limited amount of training data available. This highlights a persistent challenge in low-resource MT.
  • Impact of Manual Translations: The ablation study indicated that performance on manually created test sets (test-n3 for JPC2) could be poorer, suggesting that models trained on automatically curated data might not generalize perfectly to human-generated translations.
  • Future Work: While not explicitly detailed, the success with domain adaptation strongly suggests that further exploration of specialized corpora and advanced domain adaptation techniques would be beneficial for other challenging translation tasks. The paper also implicitly points to the need for more high-quality parallel data in low-resource Asian language pairs.

Summary updated

BibTeX
@inproceedings{park2021bering,
  author = {Heesoo Park and Dongjun Lee},
  title = {{Bering Lab’s Submissions on WAT 2021 Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure