TMU NMT System with Japanese BART for the Patent task of WAT 2021

Hwichan Kim, Mamoru KomachiACL 2021

TL;DR

This paper introduces a Neural Machine Translation (NMT) system for patent documents, leveraging a pre-trained Japanese BART model. They demonstrate that fine-tuning this Japanese BART model significantly improves translation accuracy for Korean-Japanese and English-Japanese patent tasks, even when there's little subword overlap between languages.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems perform exceptionally well when large amounts of parallel data are available. However, their performance drops significantly in low-resource scenarios, where parallel corpora are scarce. Traditional approaches to mitigate this include back-translation and transfer learning. More recently, pre-trained encoder-decoder models, trained on vast amounts of monolingual data, have shown promise in boosting NMT performance. While models like BART have demonstrated success in various NLP tasks, their effectiveness for specific language pairs, especially those with limited subword overlap and in specialized domains like patent translation, remains an open question. This paper specifically investigates the impact of a Japanese BART model on patent translation tasks involving Korean and English.

Methods

The core of the TMU NMT system is the Japanese BART (JaBART) model, which is a pre-trained encoder-decoder model. Unlike some previous approaches that used BART as an encoder or decoder separately, this work directly fine-tunes the entire JaBART model for the translation task. To adapt JaBART to the target language pairs (Korean-Japanese and English-Japanese), they initialize embeddings for unknown subwords randomly and share embeddings for matching characters across languages (like numbers and units). They also train baseline NMT models with the same architecture as JaBART for comparison. The fine-tuning and training process utilizes the fairseq implementation.

Datasets

The primary dataset used in this study is the Japan Patent Office (JPO) Corpus 2.0. This corpus provides parallel data for two language pairs:

  • Korean-Japanese (Ko-Ja)
  • English-Japanese (En-Ja)

The corpus is divided into training, development (dev), and test sets. The paper also mentions that the JaBART model itself was pre-trained on approximately 18 million Japanese Wikipedia sentences.

Key Insights and Ideas

  • Pre-trained Japanese BART is effective for patent translation: The study confirms that fine-tuning a pre-trained Japanese BART model leads to significant improvements in translation accuracy for both Korean-Japanese and English-Japanese patent tasks.
  • Subword overlap is not a prerequisite for improvement: The paper highlights that JaBART's effectiveness is observed even in language pairs like Korean-Japanese and English-Japanese, which have minimal subword overlap with Japanese. This suggests that the benefits of pre-training extend beyond languages with shared vocabulary structures.
  • Syntactic similarity might not be the sole driver of improvement: Contrary to their initial hypothesis that syntactic similarity between the pre-training language and the fine-tuning language pair would lead to greater gains, the results showed no significant difference in performance improvements between Korean-Japanese and English-Japanese. This suggests that other factors, perhaps related to the model's ability to learn general linguistic patterns, play a more dominant role.

Key Results

The fine-tuned JaBART model consistently outperformed the baseline models in both single-model and ensemble settings.

  • Single Model:
    • For Ko-Ja translation, JaBART achieved BLEU score improvements of 0.440-1.350 and RIBES score improvements of 0.001-0.007 over the baseline.
  • For En-Ja translation, JaBART showed similar gains, with BLEU score improvements of 1.013-1.250 and RIBES score improvements of 0.001-0.007.
  • Ensemble Model:
    • In Ko-Ja translations, the fine-tuned JaBART improved BLEU and RIBES scores by approximately 0.440-0.850 and 0.001-0.008, respectively.
  • In En-Ja translations, the ensemble model showed a slight decrease in BLEU score on the development set (-0.09) but an improvement on the test set (+0.240).

Overall, the fine-tuned JaBART model demonstrated consistent improvements in translation accuracy, particularly for language pairs with no subword overlap and irrespective of translation direction, except for a minor inconsistency in the En-Ja ensemble scenario.

Limitations and Future Work

  • Limitations:
    • The study found no significant difference in performance gains between language pairs with varying syntactic similarity, which contradicts their initial hypothesis. This suggests that syntactic similarity might not be the primary factor driving the effectiveness of pre-trained models in this context.
  • In the ensemble scenario for En-Ja translation, the BLEU score on the development set slightly decreased.
  • Future Work:
    • The authors suggest further investigation into other aspects, such as the training process per epoch and network representations, to better understand the behavior of fine-tuned BART models.
  • They aim to analyze BART fine-tuned using language pairs with varying syntactic proximities in more detail in the future.

Summary updated

BibTeX
@inproceedings{kim2021tmu,
  author = {Hwichan Kim and Mamoru Komachi},
  title = {{TMU NMT System with Japanese BART for the Patent task of WAT 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure