KU_ai at MEDIQA 2019: Domain-specific Pre-training and Transfer Learning for Medical NLI

Cemil Cengiz, Ulaş Sert, Deniz YuretACL 2019

TL;DR

This paper tackles the challenge of Natural Language Inference (NLI) in the medical domain, where limited data and specialized language make it difficult for standard models. The authors propose a two-stage transfer learning approach using BERT, first pre-training on general NLI datasets and then fine-tuning on the medical NLI dataset, significantly improving performance.

What Problem Does This Paper Address?

Natural Language Inference (NLI) is a fundamental AI task that involves determining the relationship (entailment, contradiction, or neutral) between two sentences, a premise and a hypothesis. While NLI has seen significant progress with large language models, applying these to specialized domains like medicine presents unique challenges. The MedNLI dataset, used in this work, is derived from clinical notes and is relatively small. Furthermore, medical text contains specialized vocabulary and de-identification artifacts that can hinder model performance. This paper aims to improve NLI performance on this challenging medical dataset by leveraging pre-trained language models and strategic transfer learning.

Methods

The core of the proposed solution is a two-stage transfer learning strategy built upon the BERT (Bidirectional Encoder Representations from Transformers) architecture.

  1. Baseline Model: The foundation is a standard BERT encoder followed by a classification head. BERT's self-attention mechanism allows it to process sentence pairs holistically, generating a joint representation.
  2. Pre-processing: To address the issue of de-identification artifacts (e.g., [**Location**], [**Name (NI)**]) in the MedNLI dataset, the authors found that removing these artifacts before tokenization significantly improved performance. This is because BERT's WordPiece tokenizer would otherwise create an excessive number of tokens for these artifacts, diluting the signal.
  3. Two-Stage Transfer Learning:
    • Stage 1 (General NLI Training): The BERT model is first fine-tuned on large, general-domain NLI datasets like MultiNLI and SNLI. This stage aims to imbue the model with general reasoning capabilities for NLI tasks.
  4. Stage 2 (Medical NLI Fine-tuning): The model from Stage 1 is then further fine-tuned on the target MedNLI dataset. This step adapts the model to the specific nuances of medical language and the clinical domain.
  5. Ensembling: To further boost performance, the authors experimented with ensembling multiple models. Specifically, they trained models using different pre-trained BERT weights (e.g., BERT BASE, BERT LARGE, BioBERT, SciBERT) and different intermediary NLI datasets (MultiNLI, SNLI) via the two-stage transfer learning process. The final predictions were made by combining the output probabilities of these individual models using element-wise operations (sum, product, max).

Datasets

  • MedNLI: The primary dataset for this shared task, consisting of clinical notes and patient records. It's characterized by its limited size and de-identification artifacts.
  • MultiNLI: A large, general-domain NLI dataset comprising written and spoken English from various genres.
  • SNLI: Another large, general-domain NLI dataset, primarily derived from image captions.

The authors used the training sets of MultiNLI and SNLI for intermediate training and the MedNLI training and development sets for fine-tuning and evaluation, respectively. The test set was reserved for the official shared task evaluation.

Key Insights and Ideas

  • Domain-Specific Pre-training Matters: Using BERT weights pre-trained on domains closer to the target task (like BioBERT and SciBERT) yielded significantly better results than general-domain BERT weights, even when the models were smaller. This highlights the importance of domain-specific knowledge.
  • De-identification Artifacts are Harmful: The presence of de-identification markers in clinical text negatively impacts BERT's performance. Removing these artifacts before tokenization was a crucial step for improving accuracy.
  • Two-Stage Transfer Learning is Effective: A sequential approach, first learning general NLI skills and then specializing in the medical domain, proved more effective than direct training on the limited MedNLI data.
  • Ensembling Combines Strengths: Combining models trained on different general-domain NLI datasets through ensembling yielded the best overall performance, suggesting that different datasets capture complementary reasoning skills.

Key Results

  • Directly training BERT on MedNLI achieved a maximum accuracy of 83.7% (with BERT LARGE).
  • The two-stage transfer learning strategy significantly improved performance. Using BioBERT with MultiNLI and SNLI for intermediate training resulted in 85.8% accuracy.
  • Ensembling models (specifically BioBERT trained with MultiNLI and SNLI) achieved the highest development set accuracy of 86.1%.
  • On the unseen test dataset for the MEDIQA 2019 NLI Shared Task, the best ensembled model achieved 84.7% accuracy, ranking 10th out of 17 teams.
  • Error analysis revealed that most misclassifications occurred between entailment and neutral labels. Ensembling helped reduce errors, particularly in the "Medical Knowledge" category.

Limitations and Future Work

  • Limited Data: The primary limitation remains the relatively small size of the MedNLI dataset, which restricts the depth of fine-tuning.
  • Future Work: The authors suggest extending BERT's capabilities by integrating external knowledge, particularly semantic relationships, to further improve performance. They also note that while their approach is strong, further improvements might come from incorporating more sophisticated reasoning mechanisms beyond standard BERT.

Summary updated

BibTeX
@inproceedings{cengiz2019ku,
  author = {Cemil Cengiz and Ulaş Sert and Deniz Yuret},
  title = {{KU\_ai at MEDIQA 2019: Domain-specific Pre-training and Transfer Learning for Medical NLI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure