MSIT_SRIB at MEDIQA 2019: Knowledge Directed Multi-task Framework for Natural Language Inference in Clinical Domain.

Sahil Chopra, Ankita Gupta, Anupama KaushikACL 2019

TL;DR

This paper introduces Bio-MTDNN, a novel framework that leverages multi-task learning and external medical knowledge (UMLS) to improve Natural Language Inference (NLI) in the clinical domain. By combining general NLU tasks with domain-specific information, Bio-MTDNN significantly outperforms existing models on the challenging clinical NLI task.

What Problem Does This Paper Address?

Natural Language Inference (NLI), the task of determining the relationship between two sentences (entailment, neutral, or contradiction), is a fundamental NLP problem. While large datasets like SNLI and MultiNLI have driven progress in general domains, the clinical domain presents unique challenges. Medical texts are complex, and patient data is sensitive, making it difficult to create large, annotated datasets. This scarcity of resources hinders the development of effective NLI models for clinical applications, which are crucial for tasks like information extraction, question answering, and document summarization in healthcare.

Methods

The authors propose Bio-MTDNN, a framework that builds upon the Multi-Task Deep Neural Network (MT-DNN) architecture. MT-DNN is designed to learn from multiple NLU tasks simultaneously, enhancing generalization. Bio-MTDNN extends this by:

  • Multi-Task Learning (MTL): Training on a variety of general NLU tasks (single-sentence classification, pairwise text classification, text similarity, relevance ranking) to learn robust representations.
  • Transfer Learning: Pre-training on large open-domain datasets and then fine-tuning on the specific clinical NLI task.
  • Domain Adaptation: Fine-tuning the model specifically for the clinical domain.
  • Knowledge Integration: Incorporating external medical knowledge from the Unified Medical Language System (UMLS), specifically the SNOMED-CT terminology. This is achieved by calculating an "interaction matrix" between premise and hypothesis tokens, where cell values represent the shortest path distance between corresponding medical concepts in the UMLS graph. This knowledge-directed representation is then integrated into the MT-DNN architecture.

They also experimented with a feature-based logistic regression classifier and a Bi-CNN-MI model for single-task transfer learning as baselines.

Datasets

  • MedNLI: The primary dataset used for evaluating the clinical NLI task. This dataset was derived from MIMIC-III and annotated by experts.
  • General Domain NLU Datasets: Used for pre-training the MT-DNN model. Specific datasets mentioned include those for paraphrase identification (PI), sentiment classification, grammatical acceptability, text similarity scoring (STS-B), and relevance ranking (QNLI).
  • UMLS (SNOMED-CT): Used as an external knowledge source.

Key Insights and Ideas

  • The Power of Multi-Task Learning for Domain Adaptation: MT-DNN's ability to learn from diverse NLU tasks provides a strong foundation for adapting to specialized domains like medicine.
  • External Knowledge is Crucial for Clinical NLI: Directly integrating medical knowledge from resources like UMLS significantly boosts performance, suggesting that general language understanding alone is insufficient for complex clinical reasoning.
  • Bi-CNN-MI Struggles with Clinical NLI: The Bi-CNN-MI model, while effective for general paraphrase identification, performed poorly on the clinical NLI task. This highlights the distinct challenges of clinical inference, where subtle differences and domain-specific relationships are critical.
  • Transfer Learning is Effective, but Domain-Specific Knowledge is Key: While transfer learning from general NLU tasks is beneficial, the true performance gains come from incorporating clinical domain knowledge.

Key Results

Bio-MTDNN achieved the highest accuracy among all evaluated systems:

  • Dev Set: 81.2% accuracy.
  • Test Set: 81.3% accuracy (submitted to the MediQA 2019 challenge).

This performance significantly outperformed:

  • MT-DNN (without external knowledge): 80.1% accuracy on the dev set.
  • Infersent: 73.5% accuracy on the dev set.
  • ESIM: 73.1% accuracy on the dev set.
  • Official Baseline: 51.9% accuracy on the dev set.
  • Feature-based Baseline: 54.1% accuracy on the dev set.
  • Bi-CNN-MI: 53.6% accuracy on the dev set.

Limitations and Future Work

  • Limitations: The paper notes that the Bi-CNN-MI model performed poorly, suggesting that knowledge gained from general paraphrase identification might not be sufficient for clinical NLI. The authors also mention that they cannot directly utilize the L1 layer embeddings from MT-DNN because they incorporate segment embeddings, which would interfere with their knowledge integration approach.
  • Future Work: The paper implicitly suggests that further exploration of knowledge integration techniques and the application of this framework to other clinical NLP tasks would be valuable. The success of Bio-MTDNN opens avenues for more sophisticated knowledge-aware models in the medical domain.

Summary updated

BibTeX
@inproceedings{chopra2019msit,
  author = {Sahil Chopra and Ankita Gupta and Anupama Kaushik},
  title = {{MSIT\_SRIB at MEDIQA 2019: Knowledge Directed Multi-task Framework for Natural Language Inference in Clinical Domain.}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure