TL;DR
This paper introduces a novel, model-independent approach to improve biomedical machine translation by enriching the training data with domain-specific terminology. By segmenting sentences to preserve biomedical terms and adding a domain feature, their system achieved impressive results, ranking 2nd and 3rd in the WMT19 Biomedical Translation Task.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) systems excel with large parallel corpora, but domain-specific translation, like in the biomedical field, often suffers from a lack of such specialized data. This scarcity leads to poor performance, particularly in terminology accuracy. Existing methods to bridge this gap often require complex model modifications or extensive domain-specific data. This paper aims to address this challenge by developing a data preparation strategy that injects domain knowledge into NMT models without altering their architecture, making it broadly applicable.
Methods
The core of this work lies in a two-pronged data preparation strategy:
- Terminology-Aware Segmentation ("bpe-terms"): Instead of standard Byte Pair Encoding (BPE) which can split important biomedical terms into subwords, this method explicitly preserves identified biomedical terms as single tokens. This is achieved by learning BPE codes but then applying them to segment sentences while ensuring that terms from a pre-defined biomedical terminology list are not broken down. This results in a mixed sequence of subword and term tokens.
- Domain Feature: A binary feature is introduced for each token, indicating whether it belongs to the biomedical terminology list or is a general domain word. This feature is then embedded into a dense vector and concatenated with the word embedding. An additional multi-layer perceptron (MLP) is applied to this combined embedding to create a richer representation that the model can leverage.
This entire process is model-independent, meaning the enriched data can be fed into any standard NMT architecture, such as the Transformer model used in this study.
Datasets
- Training Data: The authors utilized data from the WMT19 Biomedical Translation Task, including parallel sentences from sources like Scielo, UFAL, Pubmed, Medline, IBECS, and EMEA. They also incorporated monolingual domain data.
- Validation Data: The Khresmoi dataset was used for validation.
- Evaluation Data: The WMT18 and WMT19 Biomedical Translation Task test sets were used for evaluation.
- Terminology List: A list of approximately 10,000 biomedical terms for both English and Spanish was created by leveraging the BabelNet API to identify domain-specific words within their training data.
Key Insights and Ideas
- Preserving Domain Terms is Crucial: The "bpe-terms" segmentation significantly improves performance by ensuring that critical biomedical terms are treated as atomic units, preventing their fragmentation and loss of meaning.
- Domain Feature's Nuance: While the terminology-aware segmentation consistently helped, the domain feature's impact was more mixed. It seems that simply adding a binary feature might not always be beneficial and could even hurt performance in some configurations, suggesting that how this feature is integrated and learned is critical.
- Model-Agnostic Approach: The success of their data preparation strategy highlights the potential for improving NMT performance without needing to redesign the core NMT architecture, making it a versatile technique.
- BabelNet as a Powerful Resource: The use of BabelNet proved effective in extracting a substantial list of biomedical terms, forming the backbone of their domain adaptation strategy.
Key Results
The proposed systems achieved strong performance in the WMT19 Biomedical Translation Task:
- The best-performing system, which combined "bpe-terms" segmentation with shared vocabulary and embedding layers, ranked 2nd for English-Spanish (en2es) and 3rd for Spanish-English (es2en).
- In comparisons on the WMT18 test set, their "bpe-terms" segmentation approach outperformed the baseline, demonstrating the effectiveness of preserving domain terms.
- The "bpe-terms" segmentation alone showed consistent improvements over the baseline in the WMT19 task.
Limitations and Future Work
- Terminology List Dependency: The effectiveness of their approach is directly tied to the quality and coverage of the biomedical terminology list. The current list was derived from their training data, and future work aims to create a list independent of the training data.
- Multi-Word Terms: The current segmentation primarily focuses on single-word terms. Future work will explore incorporating multi-word terms extracted from biomedical glossaries.
- Domain Feature Integration: The paper notes that the domain feature approach "overall hurts the test set performances" in some configurations. Further research could explore more sophisticated ways to integrate and learn from domain features.
Summary updated