ARS_NITK at MEDIQA 2019:Analysing Various Methods for Natural Language Inference, Recognising Question Entailment and Medical Question Answering System

Anumeha Agrawal, Rosa Anil George, Selvan Suntiha Ravi, Sowmya Kamath S, Anand KumarACL 2019

TL;DR

This paper presents a system for medical question answering that integrates Natural Language Inference (NLI) and Recognizing Question Entailment (RQE) to improve information retrieval. The authors developed novel models for each task, leveraging medical domain knowledge and advanced machine learning techniques, achieving competitive results on the MediQA challenge.

What Problem Does This Paper Address?

The medical domain is awash with unstructured text data (physician notes, lab reports, etc.) that is challenging to process and utilize effectively. Intelligent Healthcare Information Management Systems (HIMS) require sophisticated natural language processing (NLP) to extract meaningful insights. Specifically, understanding the relationships between medical concepts (diseases, symptoms, treatments) is crucial. This paper tackles three related NLP challenges within this domain:

  • Natural Language Inference (NLI): Determining if a given hypothesis can be inferred from a premise statement (entailment, contradiction, or neutral).
  • Recognizing Question Entailment (RQE): Identifying if one medical question can be inferred from another, which is vital for finding answers to similar questions.
  • Medical Question Answering (QA): Filtering and ranking automatically retrieved answers to medical questions.

The core challenge is to build systems that can accurately understand the nuances of medical language and relationships to provide precise and relevant information.

Methods

The authors propose several models for each of the three tasks:

  • Natural Language Inference (NLI):
    • RNN-based Model: A baseline recurrent neural network model using GloVe embeddings.
  • InferSent Model: This model, adapted for the medical domain, uses sentence encoders to generate semantic representations. They experimented with various word embeddings (GloVe, MIMIC, Wikipedia) and crucially, integrated the UMLS Metathesaurus and NLTK WordNet synsets. UMLS Metathesaurus is a comprehensive biomedical vocabulary that links similar terms and concepts, providing rich semantic relationships. This integration significantly boosted performance.
  • Recognizing Question Entailment (RQE):
    • They developed a novel Extreme Gradient Boosting (XGBoost) model that incorporates PubMed embeddings. This approach leverages the power of ensemble learning and domain-specific embeddings to capture relationships between medical terms in questions.
  • Other models explored include Support Vector Machine (SVM), Logistic Regression, K-Nearest Neighbors (KNN), and Ada Boost, all using handcrafted lexical and semantic features, with some also incorporating PubMed embeddings.
  • Question Answering (QA):
    • A closed-domain QA system was developed using Bi-directional LSTMs trained on the SQuAD dataset. The system scrapes content from URLs, uses it as context, and then predicts answer spans. A thresholding mechanism is employed to determine answer relevance.

Datasets

  • MedNLI dataset: Used for training and evaluating NLI models. This is a publicly available expert-annotated dataset for NLI.
  • MediQA Challenge dataset: Used for experimental validation of the proposed models.
  • SQuAD dataset: Used for training the Bi-directional LSTMs for the QA task.
  • PubMed embeddings: Utilized for RQE models.
  • UMLS Metathesaurus and NLTK WordNet: Used to enrich NLI models with medical domain knowledge.

Key Insights and Ideas

  • The Power of Domain-Specific Knowledge: Integrating the UMLS Metathesaurus and NLTK WordNet significantly improved NLI performance, highlighting the importance of leveraging structured biomedical knowledge.
  • Embeddings Matter: Experimenting with various word embeddings (GloVe, MIMIC, Wikipedia) showed that domain-specific embeddings (like MIMIC) and combinations thereof can lead to better NLI results.
  • XGBoost with PubMed Embeddings for RQE: This novel approach for RQE showed promising results, suggesting that combining powerful ensemble methods with domain-specific embeddings is a fruitful direction for understanding question similarity.
  • NLI Challenges: The paper notes that predicting "neutral" labels in NLI is particularly challenging, and differentiating between entailment and neutrality requires careful handling of similar terms.

Key Results

  • NLI: The UMLS Metathesaurus and NLTK WordNet synsets model achieved the highest accuracy of 87.7% on the MediQA test dataset. InferSent with a combination of MIMIC and Wikipedia embeddings reached 74.4%.
  • RQE: The XGBoost model with PubMed embeddings achieved the highest accuracy of 66.7% on the test set.
  • QA: The closed-domain QA model achieved an accuracy of 53.6%, precision of 55.9%, and Mean Reciprocal Rank of 62.93%. This was noted as being significantly above the baseline of 51%.

Limitations and Future Work

  • NLI Error Analysis: The authors observed that predicting "neutral" labels is difficult, and differentiating between entailment and neutrality is challenging. They also noted that while similar terms might suggest entailment, the statements could actually be neutral.
  • Future Work:
    • Extend the textual inference model for the clinical domain to develop decision support applications that simplify treatment methods by grouping similar diseases.
  • Improve QA accuracy by training on the MedQuAD dataset.
  • Explore Bidirectional LSTMs or GRUs to potentially improve NLI accuracy further.

Summary updated

BibTeX
@inproceedings{agrawal2019ars,
  author = {Anumeha Agrawal and Rosa Anil George and Selvan Suntiha Ravi and Sowmya Kamath S and Anand Kumar},
  title = {{ARS\_NITK at MEDIQA 2019:Analysing Various Methods for Natural Language Inference, Recognising Question Entailment and Medical Question Answering System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure