Abstract
LMU Munich's participation in the WMT 2018 biomedical translation task highlights the effectiveness of the Transformer architecture and specialized domain adaptation techniques. By combining linguistically informed preprocessing—specifically for German morphology—with incremental training and right-to-left reranking, their systems achieved significant improvements over previous state-of-the-art models. Notably, the Transformer architecture outperformed heavily optimized GRU-based models, establishing a new baseline for medical NMT.
Introduction
Machine Translation (MT) models often struggle when transitioned from general domains, such as news or parliamentary proceedings, to specialized fields like medicine. Medical texts present unique challenges including a high density of technical terminology, context-specific sense shifts for common words, and complex morphology. In German, for instance, compound words like Gesundheitsinformationsblatt can be difficult for standard subword tokenization methods to process effectively.
This research explores bridging the domain gap through meticulous domain adaptation, specifically investigating how to effectively fine-tune models using a strategic mix of general-purpose and high-quality in-domain data.
Methodology
Neural Architectures
The researchers evaluated two primary neural architectures:
- GRU Encoder-Decoder: Utilizing the Nematus toolkit, they developed a single-hidden-layer architecture optimized through a rigorous incremental training regime.
- Transformer: Using the Sockeye implementation, they trained "big" (En→De) and "small" (De→En) models. Unlike the GRU models, the Transformer was trained directly on the final data mix without complex incremental steps.
Linguistically Informed Preprocessing
To address the complexities of German morphology, a cascaded preprocessing pipeline was implemented before applying Byte Pair Encoding (BPE):
- Suffix Splitting: Separating morphological suffixes from stems.
- Compound Splitting: Deconstructing German compound nouns into their constituent parts.
- BPE: Applying subword tokenization to the already-split components to manage the remaining vocabulary.
Incremental Training and Reranking
For GRU models, the team used successive optimization runs, starting with large general datasets and gradually refining the model on smaller, higher-quality medical datasets. Additionally, they employed $n$-best list reranking ($n=50$) using a "right-to-left" (r2l) model, which scores the likelihood of target sentences in reverse order to provide a final performance boost.
Experimental Setup
The models were trained on a combination of general domain data (Europarl, News Commentary, Common Crawl) and medical domain data (UFAL medical corpus and HimL project data). Evaluation was conducted on the Cochrane and NHS24 portions of the HimL devtest set, representing professional research and consumer health leaflets, respectively.
Results and Discussion
The results indicate that while meticulous tuning is beneficial, architecture choice is paramount. A key finding was that a relatively standard Transformer model outperformed a GRU model that had undergone extensive multi-stage fine-tuning. This suggests the Transformer's attention mechanism is inherently more efficient at capturing the long-range dependencies and technical context required for medical translation.
As detailed in Figure 1, the incremental addition of data and the application of r2l reranking consistently improved BLEU scores. The Transformer ensemble + r2l achieved the best results, with 40.3 BLEU on Cochrane and 35.5 on NHS24, a substantial improvement over the 35.8 and 30.3 scores of the LMU WMT17 baseline.
In the German→English direction, the Transformer ensemble reached 54.5 BLEU on the Cochrane set, demonstrating that the architecture scales exceptionally well when the target language has simpler morphology.
Conclusion
The LMU Munich systems demonstrate that the Transformer architecture, combined with linguistically-informed preprocessing and r2l reranking, sets a high bar for biomedical translation. Future work may involve training native right-to-left Transformers for reranking and exploring deeper linguistic analysis, such as lemma-based tagging, to further refine the handling of rich German morphology.
Summary updated