Abstract
In this paper, we describe the neural machine translation (NMT) system submitted by the Kangwon National University and HYUNDAI (KNU-HYUNDAI) team to the translation tasks of the 6th workshop on Asian Translation (WAT 2019). We participated in all tasks of ASPEC and JPC2, which included those of Chinese-Japanese, English-Japanese, and Korean-Japanese. We submitted our transformer-based NMT system built using the following methods: a) relative positioning method for pairwise relationships between the input elements, b) back-translation and multi-source translation for data augmentation, c) right-to-left (r2l)-reranking model robust against error propagation in autoregressive architectures such as decoders, and d) checkpoint ensemble models, which selected the top three models with the best validation bilingual evaluation understudy (BLEU). We performed well in both tasks and were ranked first in terms of the BLEU scores in all the JPC2 subtasks we participated in.
Introduction
Machine translation for scientific papers and patents is notoriously difficult due to highly specialized terminology, complex sentence structures, and the "open vocabulary" problem where rare technical terms are absent from training data. Furthermore, standard Neural Machine Translation (NMT) models are autoregressive, generating text sequentially. If a model makes an error early in a sentence, that error propagates, often compromising the entire translation. The KNU-HYUNDAI team sought to build a system more structurally aware and robust against these sequential errors for the WAT 2019 competition.
Methodology
The team built their system on the Transformer architecture via the OpenNMT toolkit, introducing several key enhancements to improve technical translation quality:
- Relative Position Representation: Instead of absolute positioning, this method models the pairwise relationships between input elements. This helps the model better understand the internal structure of complex technical sentences.
- Data Augmentation (Back-translation & Multi-source): To expand training data, the team used back-translation (translating monolingual target text back into the source language). They also employed multi-source translation, training the model on multiple source languages (e.g., English and Korean) mapping to the same target language (Japanese).
- Right-to-Left (R2L) Reranking: This serves as a "sanity check" for the decoder. The system generates several candidate translations using standard left-to-right methods, which are then scored by a separate model trained to read from right-to-left. This promotes translations that are globally coherent.
- Checkpoint Ensembling: The team averaged the top three models based on validation performance to smooth out variance and improve final accuracy.
Experimental Setup
The models were trained and evaluated on two primary benchmarks:
- ASPEC (Asian Scientific Paper Excerpt Corpus): Focused on English-Japanese and Chinese-Japanese scientific abstracts.
- JPC2 (Japan Patent Office Patent Corpus 2.0): Focused on English-Japanese, Chinese-Japanese, and Korean-Japanese patent documents.
Key Results and Analysis
The KNU-HYUNDAI system was highly successful, particularly in the patent domain, ranking 1st in all JPC2 subtasks entered (En-Ja, Ja-En, Zh-Ja, Ja-Zh, and Ko-Ja).
Ablation Success
Ablation studies showed a clear cumulative benefit for every feature added. For the ASPEC English-to-Japanese task, the baseline BLEU of 40.34 jumped to 44.08 after adding relative positioning, back-translation, ensembling, and R2L reranking.
In the JPC2 domain, the team achieved a peak BLEU of 47.38 for English-to-Japanese and 44.72 for Japanese-to-English. The inclusion of ASPEC data as an additional training source for JPC2 tasks also provided a notable performance boost.
Domain-Specific Insights
- Structure and Synergy: The success of relative positioning suggests that for technical text, understanding the relationship between words is more important than their simple index in a sequence. Furthermore, multi-source augmentation showed that data from one language pair can improve another when the target language is shared.
- Linguistic Nuance: The team noted that multi-source benefits depend on linguistic distance. For the Korean-to-Japanese task, multi-source training (70.62 BLEU) slightly underperformed compared to a model using only relative positioning (71.52 BLEU), though the final system still achieved a remarkably high BLEU score of 73.04 for the Ko-Ja task overall.
Conclusion
The KNU-HYUNDAI team's performance at WAT 2019 demonstrates that combining structural model improvements with sophisticated reranking and data augmentation is highly effective for technical domains. By addressing the weaknesses of autoregressive decoding, the system set a new benchmark for patent and scientific translation.
Summary updated