Exploring Transfer Learning and Domain Data Selection for the Biomedical Translation

Noor-e-Hira, Sadaf Abdul Rauf, Kiran Kiani, Ammara Zafar, Raheel NawazACL 2019

TL;DR

This paper tackles the challenge of improving Neural Machine Translation (NMT) for the specialized domain of biomedical texts, which often suffer from limited parallel data. The authors demonstrate that combining transfer learning with a novel selective data training approach, using information retrieval to pinpoint relevant out-of-domain sentences, significantly boosts translation quality compared to standard methods.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized machine translation, but its performance often degrades when applied to specialized domains like biomedicine. This is primarily due to the scarcity of large, high-quality parallel corpora (texts translated into multiple languages) in these niche areas. While general-domain NMT systems perform well on news or general text, they struggle with the unique terminology and sentence structures found in medical literature. Existing approaches like domain adaptation and transfer learning show promise, but effectively leveraging available out-of-domain data to improve biomedical translation remains a significant challenge. This paper specifically addresses how to best utilize out-of-domain data for biomedical NMT by exploring selective data training and transfer learning.

Methods

The core of this work lies in two main techniques:

  • Transfer Learning: This involves initializing a new NMT model with parameters from a pre-trained model. The authors leverage this by using a general-domain NMT system as a starting point for training their biomedical models.
  • Selective Data Training (with Information Retrieval): Instead of using all available out-of-domain data, the authors employ an Information Retrieval (IR) approach. They use sentences from the in-domain biomedical corpus (specifically Medline titles) as queries to search a large out-of-domain corpus (News Commentary). The top-ranked, most relevant sentences from the out-of-domain data are then selected and used for additional training. This aims to enrich the training data with domain-relevant examples without the computational cost of using the entire out-of-domain corpus.

The paper also investigates the impact of tokenization, a pre-processing step that breaks down text into smaller units (like sub-word units), on NMT performance. They compare models trained on tokenized versus untokenized data.

Datasets

The authors utilized a combination of in-domain and out-of-domain corpora:

  • In-domain Corpora: EMEA, Scielo, UFAL Medical Corpus, Medline Abstracts, and Medline Titles.
  • Out-domain Corpora: News Commentary (used for IR to select relevant sentences) and Books corpus.

The Medline Titles corpus was specifically used as the test set for evaluating the models. The News Commentary corpus was used for both general domain training and as the source for selective data retrieval.

Key Insights and Ideas

  • Targeted Data is Key: Simply adding a large amount of general-domain data (like the full News Commentary corpus) to an in-domain model provides only a modest improvement. However, selectively choosing domain-relevant sentences from that general corpus using IR yields significantly better results.
  • IR as a Smart Filter: The IR-based selective data training approach is highly effective, demonstrating that a smaller, carefully curated set of out-of-domain sentences can be more beneficial than a much larger, unselected set.
  • Tokenization Matters (A Lot): The study strongly emphasizes the importance of tokenization for NMT. Models trained on tokenized data consistently outperform those trained on untokenized data, with performance drops of several BLEU points when tokenization is omitted.
  • Cascaded Transfer Learning Works: Applying transfer learning sequentially—first to a general domain, then to an out-of-domain corpus, and finally to in-domain data—can lead to substantial gains, especially when starting from a weak out-domain baseline.

Key Results

  • The best model achieved a significant improvement of 17.44 BLEU points over the out-domain books corpus baseline by incorporating all in-domain data.
  • Selective data training using IR improved performance by 4.11 BLEU points when initializing from an in-domain baseline and adding selected out-of-domain data.
  • When starting from an out-domain baseline (Books corpus), adding in-domain data drastically improved system performance, giving a gain of around 10 BLEU points.
  • The most effective model for English-to-French translation, combining cascaded transfer learning and selective data training, achieved 21.97 BLEU points.
  • On average, untokenized corpora led to a performance decline of around 4 BLEU points compared to tokenized corpora.

Limitations and Future Work

  • Limited Scope of IR: While effective, the IR approach was limited to selecting sentences from the News Commentary corpus. Future work could explore IR across a wider range of out-of-domain sources.
  • Tokenization Impact: The paper highlights the importance of tokenization but doesn't delve into optimizing specific tokenization strategies for biomedical texts. Further research could explore advanced tokenization techniques tailored to this domain.
  • French-to-English Direction: The French-to-English direction used a simpler training pipeline. Exploring the same selective data training and transfer learning strategies for this direction could yield further improvements.
  • Domain Specificity: While the focus is biomedical, the generalizability of these selective data training and transfer learning techniques to other specialized domains could be investigated.

Summary updated

BibTeX
@inproceedings{noorehira2019exploring,
  author = {Noor-e-Hira and Sadaf Abdul Rauf and Kiran Kiani and Ammara Zafar and Raheel Nawaz},
  title = {{Exploring Transfer Learning and Domain Data Selection for the Biomedical Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure