Saama Research at MEDIQA 2019: Pre-trained BioBERT with Attention Visualisation for Medical Natural Language Inference

Kamal raj Kanakarajan, Suriyadeepan Ramamoorthy, Vaidheeswaran Archana, Soham Chatterjee, Malaikannan SankarasubbuACL 2019

TL;DR

This paper introduces a BioBERT model, pre-trained on biomedical literature and fine-tuned on clinical notes, to tackle Natural Language Inference (NLI) in the medical domain (MedNLI). The model achieves state-of-the-art results on MedNLI and demonstrates the effectiveness of attention visualization in understanding model behavior.

What Problem Does This Paper Address?

Natural Language Inference (NLI) is a fundamental NLP task that aims to determine the relationship between two sentences: entailment (hypothesis is true given the premise), contradiction (hypothesis is false), or neutral (neither). While general NLI datasets like SNLI exist, the medical domain presents unique challenges due to specialized terminology and the critical need for accurate inference in clinical settings. Existing models struggled to achieve high performance on medical NLI tasks. This paper addresses the need for a robust NLI model specifically for the clinical domain, aiming to improve accuracy and provide insights into how such models work.

Methods

The core of this work is the application of BERT (Bidirectional Encoder Representations from Transformers), a powerful language representation model. Specifically, the authors utilize BioBERT, a BERT model pre-trained on a large corpus of biomedical literature (PubMed and PMC). They then fine-tune this BioBERT model on the MIMIC-III v1.4 dataset, a collection of de-identified clinical notes. This fine-tuning step adapts the general biomedical knowledge of BioBERT to the specific nuances of clinical text.

A key aspect of their methodology is the use of attention visualization tools, specifically bertviz, to analyze the attention patterns within the BioBERT model. This allows them to gain insights into how the model processes sentence relationships before and after fine-tuning, helping to understand the model's decision-making process.

Datasets

  • MedNLI: A biomedical NLI dataset analogous to SNLI, designed for the clinical domain.
  • MIMIC-III v1.4: A large dataset of de-identified clinical notes used for fine-tuning the BioBERT model.
  • MEDIQA: A shared task dataset used as an additional evaluation benchmark for the proposed model.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: Pre-training BERT on biomedical literature (BioBERT) significantly improves performance on medical NLI tasks compared to general-domain BERT.
  • Fine-tuning on Clinical Data Matters: Further fine-tuning BioBERT on clinical notes (MIMIC-III v1.4) is essential to adapt the model to the specific language and context of clinical NLI.
  • Attention Patterns Reveal Learning: Visualizing attention patterns before and after fine-tuning provides valuable insights into how the model learns to establish connections between premise and hypothesis sentences. For instance, before fine-tuning, attention is often focused on the [SEP] token, while after fine-tuning, it becomes more distributed across the entire hypothesis sentence, indicating a better understanding of sentence relationships.
  • Word Similarity is Key for NLI: The model's ability to identify and connect semantically similar words, even when expressed differently (e.g., abbreviations vs. full terms), is critical for accurate NLI.
  • Handling Abbreviations is Important: The paper highlights that the model's difficulty in recognizing abbreviations (like "n/v") and their expanded forms ("nausea and vomiting") can lead to misclassifications, suggesting preprocessing steps to normalize such terms.

Key Results

  • The proposed BioBERT model, pre-trained on PubMed and PMC and fine-tuned on MIMIC-III v1.4, achieved state-of-the-art results on the MedNLI test set with an accuracy of 83.45%.
  • The same model achieved an accuracy of 78.5% on the MEDIQA test set.
  • The authors observed distinct attention patterns after fine-tuning, showing a more distributed and dense connection between sentences, which they attribute to the NLI task's requirement for establishing connections.

Limitations and Future Work

  • Lack of Text Preprocessing: The paper acknowledges that the only preprocessing step used was tokenization. For domain-specific tasks like medical NLI, identifying and normalizing medical concepts (which can have multiple representations) is crucial.
  • Vocabulary Limitations: The current vocabulary is derived from Wikipedia, leading to many common medical terms being identified as unknown words and split into tokens.
  • Future Work Suggestions:
    • Implement a preprocessing step to expand and normalize abbreviated medical terms.
  • Build a vocabulary based on a PubMed corpus to ensure common medical terms are recognized.
  • Explore the use of entity embeddings to learn and leverage medical concepts more effectively.

Summary updated

BibTeX
@inproceedings{kanakarajan2019saama,
  author = {Kamal raj Kanakarajan and Suriyadeepan Ramamoorthy and Vaidheeswaran Archana and Soham Chatterjee and Malaikannan Sankarasubbu},
  title = {{Saama Research at MEDIQA 2019: Pre-trained BioBERT with Attention Visualisation for Medical Natural Language Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure