LMU Munich's Neural Machine Translation Systems at WMT 2018

Matthias Huck, Dario Stojanovski, Viktor Hangya, Alexander M. FraserWMT (shared Task) 2018 · DOI · OpenReview

Abstract

LMU Munich's participation in the WMT 2018 biomedical translation task highlights the effectiveness of the Transformer architecture and specialized domain adaptation techniques. By combining linguistically informed preprocessing—specifically for German morphology—with incremental training and right-to-left reranking, their systems achieved significant improvements over previous state-of-the-art models. Notably, the Transformer architecture outperformed heavily optimized GRU-based models, establishing a new baseline for medical NMT.

Introduction

Machine Translation (MT) models often struggle when transitioned from general domains, such as news or parliamentary proceedings, to specialized fields like medicine. Medical texts present unique challenges including a high density of technical terminology, context-specific sense shifts for common words, and complex morphology. In German, for instance, compound words like Gesundheitsinformationsblatt can be difficult for standard subword tokenization methods to process effectively.

This research explores bridging the domain gap through meticulous domain adaptation, specifically investigating how to effectively fine-tune models using a strategic mix of general-purpose and high-quality in-domain data.

Methodology

Neural Architectures

The researchers evaluated two primary neural architectures:

  • GRU Encoder-Decoder: Utilizing the Nematus toolkit, they developed a single-hidden-layer architecture optimized through a rigorous incremental training regime.
  • Transformer: Using the Sockeye implementation, they trained "big" (En→De) and "small" (De→En) models. Unlike the GRU models, the Transformer was trained directly on the final data mix without complex incremental steps.

Linguistically Informed Preprocessing

To address the complexities of German morphology, a cascaded preprocessing pipeline was implemented before applying Byte Pair Encoding (BPE):

  1. Suffix Splitting: Separating morphological suffixes from stems.
  2. Compound Splitting: Deconstructing German compound nouns into their constituent parts.
  3. BPE: Applying subword tokenization to the already-split components to manage the remaining vocabulary.

Incremental Training and Reranking

For GRU models, the team used successive optimization runs, starting with large general datasets and gradually refining the model on smaller, higher-quality medical datasets. Additionally, they employed $n$-best list reranking ($n=50$) using a "right-to-left" (r2l) model, which scores the likelihood of target sentences in reverse order to provide a final performance boost.

Experimental Setup

The models were trained on a combination of general domain data (Europarl, News Commentary, Common Crawl) and medical domain data (UFAL medical corpus and HimL project data). Evaluation was conducted on the Cochrane and NHS24 portions of the HimL devtest set, representing professional research and consumer health leaflets, respectively.

Results and Discussion

The results indicate that while meticulous tuning is beneficial, architecture choice is paramount. A key finding was that a relatively standard Transformer model outperformed a GRU model that had undergone extensive multi-stage fine-tuning. This suggests the Transformer's attention mechanism is inherently more efficient at capturing the long-range dependencies and technical context required for medical translation.

BLEU scores for English→German translation across various training stages and architectures.

As detailed in Figure 1, the incremental addition of data and the application of r2l reranking consistently improved BLEU scores. The Transformer ensemble + r2l achieved the best results, with 40.3 BLEU on Cochrane and 35.5 on NHS24, a substantial improvement over the 35.8 and 30.3 scores of the LMU WMT17 baseline.

In the German→English direction, the Transformer ensemble reached 54.5 BLEU on the Cochrane set, demonstrating that the architecture scales exceptionally well when the target language has simpler morphology.

Conclusion

The LMU Munich systems demonstrate that the Transformer architecture, combined with linguistically-informed preprocessing and r2l reranking, sets a high bar for biomedical translation. Future work may involve training native right-to-left Transformers for reranking and exploring deeper linguistic analysis, such as lemma-based tagging, to further refine the handling of rich German morphology.

Summary updated

Abstract

We present the LMU Munich machine translation systems for the English–German language pair. We have built neural machine translation systems for both translation directions (English→German and German→English) and for two different domains (the biomedical domain and the news domain). The systems were used for our participation in the WMT18 biomedical translation task and in the shared task on machine translation of news. The main focus of our recent system development efforts has been on achieving improvements in the biomedical domain over last year’s strong biomedical translation engine for English→German (Huck et al., 2017a). Considerable progress has been made in the latter task, which we report on in this paper.
BibTeX
@inproceedings{huck2018lmu,
  author = {Matthias Huck and Dario Stojanovski and Viktor Hangya and Alexander M. Fraser},
  title = {{LMU Munich's Neural Machine Translation Systems at WMT 2018}},
  booktitle = {WMT (shared Task)},
  year = {2018},
  doi = {10.18653/v1/w18-6446},
}
Figure