IIT-KGP at MEDIQA 2019: Recognizing Question Entailment using Sci-BERT stacked with a Gradient Boosting Classifier

Prakhar Sharma, Sumegh RoychowdhuryACL 2019

TL;DR

This paper introduces QSpider, a novel system for recognizing question entailment in the medical domain. By combining the power of Sci-BERT for semantic understanding with a Gradient Boosting Classifier that leverages question type features, QSpider achieves state-of-the-art results on the MEDIQA 2019 task, outperforming previous methods. This work is crucial for improving the efficiency and accuracy of health-related question-answering systems.

What Problem Does This Paper Address?

The internet is a primary source for health information, but the sheer volume of data and the way questions are phrased can make it difficult for users to find relevant and trustworthy answers. Duplicate or semantically similar questions complicate information retrieval. This paper tackles the problem of "question entailment" in the medical domain, specifically aiming to determine if the answer to one medical question implies the answer to another. This is particularly challenging because medical questions can be phrased in many ways, using synonyms or abbreviations, making simple keyword matching insufficient. The goal is to build systems that can understand these nuances and connect related questions, thereby improving the retrieval of answers from existing FAQs or knowledge bases.

Methods

The core of the QSpider system is a two-stage approach. First, it utilizes Sci-BERT, a domain-specific BERT model pre-trained on scientific and biomedical corpora, to capture rich semantic representations of the input questions. This is a significant improvement over general-purpose language models as it's tailored for the medical domain. The output from Sci-BERT is then fed into a Gradient Boosting Classifier.

What makes QSpider novel is its incorporation of question type prediction as a feature for the classifier. The authors trained a multi-label classifier to identify one of 13 question types (e.g., "Information," "Diagnosis," "Anatomy") from consumer health questions. These predicted question types, represented as a binary vector, are then horizontally stacked with the Sci-BERT embeddings of the two questions. This combined feature vector is used by the Gradient Boosting Classifier to make the final entailment prediction. This approach allows the system to consider both the semantic content of the questions and their underlying intent.

Datasets

The primary dataset used is the MEDIQA 2019 RQE Shared Task dataset. This dataset consists of pairs of consumer health questions (CHQs) and frequently asked questions (FAQs). The training set contains 8,588 pairs, with 54.2% being positive (entailment) examples. Negative examples were generated by pairing questions with similar keywords but different meanings. The validation set has 302 pairs, and the hidden test set has 230 pairs.

Additionally, the authors used an annotated corpus of consumer health questions from Roberts et al. (2014) to build their question type prediction classifier. This corpus contains 1,467 consumer-generated requests classified into 13 question types.

Key Insights and Ideas

  • Domain-specific pre-training matters: Using Sci-BERT, which is pre-trained on biomedical text, significantly outperforms general BERT models and other approaches on this medical question entailment task.
  • Question types are valuable features: Incorporating predicted question types as features for the entailment classifier provides crucial contextual information that aids in distinguishing between related and unrelated questions, especially when medical entities are phrased differently.
  • Fine-grained entailment: The approach allows for a more nuanced understanding of entailment, moving beyond simple lexical overlap to capture semantic relationships.
  • Handling synonyms and abbreviations: The combination of Sci-BERT's semantic understanding and question type features helps in correctly identifying entailment even when medical entities are expressed as synonyms or abbreviations.

Key Results

QSpider achieved an accuracy of 68.4% on the hidden test set of the MEDIQA 2019 task, making it the best performing system among the participating models and ranking 3rd overall. This represents a significant improvement over the baseline model (54.1%) and other strong contenders. For instance, on the validation set, QSpider achieved 62.0% accuracy, while Sci-BERT + Hinge Loss achieved 80.5% and BERTlarge achieved 76.2%. However, QSpider's performance on the test set (68.4%) surpassed BERTlarge (48.1%) and Bio-BERT (49.6%), highlighting its robustness.

Limitations and Future Work

The authors acknowledge that the training data for question type prediction was small, which necessitated the use of pre-trained language models to learn effectively. They also note that their system might misclassify questions where the medical entities are different but the questions have the same type.

Future work includes:

  • Integrating medical entity detection: Appending detected medical entity names to the feature vector to better handle difficult examples where entities differ.
  • Improving question type representation: Moving beyond discrete classes to use GloVe embeddings of question classes as extended features for the classifier, acknowledging that questions can often blend multiple types.

Summary updated

BibTeX
@inproceedings{sharma2019iit,
  author = {Prakhar Sharma and Sumegh Roychowdhury},
  title = {{IIT-KGP at MEDIQA 2019: Recognizing Question Entailment using Sci-BERT stacked with a Gradient Boosting Classifier}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure