TL;DR
This paper details the UU_TAILS team's participation in the MEDIQA 2019 challenge, focusing on improving Natural Language Inference (NLI) and Recognizing Textual Entailment (RQE) models for the medical domain. They achieved strong results by leveraging BERT embeddings and sophisticated ensemble techniques for NLI, and Universal Sentence Encoder (USE) embeddings for RQE.
What Problem Does This Paper Address?
Accurately understanding the semantic relationship between sentences is a cornerstone of Natural Language Processing (NLP). Textual Entailment Recognition (TER) aims to determine if the meaning of one sentence (hypothesis) can be inferred from another (premise). While general-domain NLI has seen significant progress with large datasets like SNLI and MultiNLI, adapting these models to specialized domains like medicine presents a significant challenge. Medical language is often complex, filled with domain-specific jargon, and can have subtle semantic nuances that general models struggle to grasp. The MEDIQA 2019 challenge specifically targeted these issues by introducing domain-specific tasks and datasets for NLI and RQE.
Methods
For the Natural Language Inference (NLI) task, the UU_TAILS team explored various approaches centered around BERT embeddings. They experimented with different BERT models, including SciBERT and ClinicalBERT, which were pre-trained on scientific and clinical text respectively. To boost performance, they employed several ensemble techniques:
- Drop-out Averaging: Iteratively dropping models from an ensemble and averaging probabilities to find the optimal subset.
- Stacking BERT 1: Using meta-learners (KNN, SVM, Naive Bayes) trained on the predictions of base BERT models.
- Stacking BERT 2: A second-level ensemble where a logistic regression classifier was trained on the outputs of the first-level stacking models.
For the Recognizing Question Entailment (RQE) task, which involves identifying duplicate questions based on semantic similarity, they utilized the Universal Sentence Encoder (USE). They embedded input questions separately and then combined these embeddings using concatenation, element-wise product, and absolute difference. These combined representations were then fed into either a logistic regression classifier or a Multi-Layer Perceptron (MLP).
Datasets
- MedNLI: A medical Natural Language Inference benchmark dataset created by adapting the SNLI dataset to the clinical domain. It contains 14,049 sentence pairs.
- MEDIQA 2019 Challenge Datasets: The challenge provided specific datasets for its three tasks: NLI, RQE, and Question Answering. The paper focuses on the NLI and RQE tasks.
- RQE Dataset: Constructed from questions provided by family doctors on the National Library of Medicine (NLM) platform, containing 8,588 question pairs for training.
Key Insights and Ideas
- Domain-Specific Pre-training Matters: Using BERT models pre-trained on medical text (SciBERT, ClinicalBERT) provided a significant advantage over general-domain models for medical NLI.
- Ensembling is Crucial for Performance: Sophisticated ensemble methods, particularly multi-level stacking, were key to achieving top performance in the NLI task, demonstrating that combining diverse model predictions can overcome individual model limitations.
- Transfer Learning Challenges: While transfer learning was explored for RQE (using a model trained on MedNLI), it did not yield improvements, suggesting that direct training on domain-specific RQE data or different transfer learning strategies might be more effective.
- USE for RQE: The Universal Sentence Encoder proved to be a strong baseline for the RQE task, highlighting its ability to capture sentence semantics effectively for similarity-based tasks.
Key Results
- NLI Task: The UU_TAILS team achieved a test set accuracy of 0.852 for the NLI task. Their best performing submission ranked 12th out of 42 teams. Their top model was a three-stage ensemble architecture with two levels of stacking.
- RQE Task: The team achieved a test set accuracy of 0.584 for the RQE task, ranking 23rd out of 54 participants. The USE embeddings with an MLP classifier yielded slightly better results than with logistic regression.
Limitations and Future Work
- NLI Future Work: The authors suggest exploring BERT in a feature extraction mode rather than fine-tuning the entire model. This would allow for more flexibility in selecting and combining embeddings from different layers.
- RQE Future Work: The team aims to investigate other inferSent architectures and explore training on clinical word embeddings for the RQE task. They also acknowledge that more extensive hyperparameter tuning for the MLP classifier could potentially improve results.
- General Limitation: The paper notes that the RQE task was approached with a simpler model due to time constraints, implying that more complex architectures might yield better performance.
Summary updated